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Abstract 

This  research  used  systems  architecture  to  develop  a  model  that  determined  the  effect  of 
Integrated  System  Health  Management  (ISHM)  on  mission  success  rates  for  unmanned 
aerial  systems  (UAS).  To  evaluate  this  effect,  a  simulation  model  was  developed  and 
used  to  analyze  the  difference  between  mission  success  rates  for  a  theoretical  UAS  with 
and  without  ISHM.  Design  of  Experiments  analysis  techniques  were  used  to  map  a 
response  surface  that  modeled  the  difference  between  mission  success  rates  calculated  for 
current  health  management  technology  and  ISHM.  Using  representative  data  for  a  UAS, 
the  analysis  determined  that  the  failure  distribution  parameters,  sensor  quality  (which 
determines  the  relationship  between  probability  of  detection  and  probability  of  false 
alarm),  and  probability  of  an  imminent  fault  during  a  mission  were  significant  to  the 
model.  The  result  of  the  model  determined  that  ISHM  can  result  in  a  significant 
improvement  on  mission  assurance,  especially  when  implemented  with  higher  quality 
sensors  and  on  vehicles  where  the  probability  of  imminent  failure  is  higher  relative  to  the 
mission  times  and  time  between  preventative  maintenance.  This  appears  consistent  with 
the  premise  that  ISHM  can  support  an  extension  of  preventative  maintenance  intervals 
with  an  attendant  reduction  in  sustainment  cost. 
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EVALUATING  THE  EFFECT  OF  INTEGRATED  SYSTEM  HEALTH 
MANAGEMENT  ON  MISSION  EFFECTIVENESS 


I.  Introduction 


1.1  Background 

In  2010,  the  United  States  Air  Force  (USAF)  released  the  results  of  a  year-long  study 
highlighting  the  need  for  increasing  autonomy  in  modem  weapon  systems,  especially  in 
the  domain  of  unmanned  aerial  systems  (UAS).  The  study  identified  the  need  for  greater 
system  autonomy  as  the  “single  greatest  theme”  for  future  USAF  Science  and 
Technology  investments  [Dahm,  2010].  Current  technology  advancements  have  brought 
the  USAF  to  a  state  of  flexible  autonomy,  which  involves  dynamically  shifting  command 
and  control  (C2)  from  autonomous  to  operator  based  on  workload,  system  health,  and  the 
perceived  intent  of  the  operator. 

One  of  the  key  attributes  sustaining  flexible  autonomy  is  the  ability  of  the  UAS  to  self- 
detect,  isolate,  and  diagnose  system  health  problems.  Current  flight  avionics  architectures 
may  include  lower  level  sub-system  health  monitoring  or  may  isolate  health  monitoring 
functions  to  a  black  box  configuration,  but  a  vehicle- wide  health  monitoring  information 
system  has  seldom  been  implemented.  A  new  area  of  research,  Integrated  System  Health 
Management  (ISHM),  adds  a  centralized  health  management  system  that  is  responsible 
for  collecting  and  processing  vehicle  health  status  information  from  across  the  vehicle 
during  all  mission  phases.  ISHM  balances  data  flow  from  multiple  sub-systems  and 
produces  the  information  necessary  to  identify  current  vehicle  capabilities,  provide 
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situational  awareness  to  mission  and  ground  operations,  and  quickly  identify 
contingencies  for  improved  vehicle  control  and  mission  decisions. 


1.2  Problem  Statement 

Although  reliability  has  improved  since  the  last  official  UAS  reliability  study,  the 
Unmanned  Aerial  Vehicle  Reliability  Study  commissioned  by  the  Office  of  the  Secretary 
of  Defense  in  2003,  significant  problems  still  plague  the  overall  health  of  the  systems. 
Current  statistics  on  the  loss  rate  per  100,000  flight  hours  of  several  unmanned  systems 
are  compared  with  various  manned  military  aircraft  in  Table  1.  The  UAS  loss  rates  are 
magnitudes  above  the  manned  aircraft,  although  some  UAS  platforms  have  not  yet 
reached  100,000  lifetime  flight  hours. 


Table  1  -  Class  A  Mishap  Rates  per  100,000  Flight  Hours  [AF  Safety  Center,  2012] 


UAS 

Mishap  Rate  Per 
100K  Hours 

Manned 

Aircraft 

Mishap  Rate  Per 
100K  Hours 

Predator 

7.69 

F-16 

3.58 

Global  Hawk* 

11.37 

B-52 

1.29 

Reaper 

6.37 

C-5 

1.04 

*  Has  not  reached  100k  flight  hours 

C-130 

0.83 

The  dominant  causes  of  these  UAS  mishaps  are  presented  in  Table  2. 


Table  2  -  Causes  of  [UAS]  Mishaps  [OUSDATL,  2004] 


[UAS]  Mishap  Cause 

Percent 

Power  and  Propulsion 

37% 

Flight  Controls 

25% 

Human  Error 

17% 

Communications 

11% 

Miscellaneous 

10% 

The  two  mishap  causes  where  there  can  be  an  assumption  made  that  the  current  health 

management  or  monitoring  system  did  not  adequately  detect  an  imminent  failure  are 
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Power  and  Propulsion  and  Flight  Controls,  which  amount  to  62%  of  total  mishap  causes. 
Granted,  even  a  theoretically  ideal  health  management  system  cannot  account  for  every 
fault  or  failure  cause,  but  vast  improvements  need  to  be  made  in  fault  detection  systems. 

There  is  also  cause  for  concern  on  the  maintenance  side  of  health  management.  Even 
when  a  fault  is  detected  pervasive  “Could  Not  Duplicate”  (CND)  and  “No  Defect  Found” 
(NDF)  maintenance  results  show  that  improvement  in  fault  isolation  is  needed.  In  1999, 
an  average  of  nine  CND  and  47  NDF  maintenance  results  were  recorded  per  aircraft 
[Stoll,  2000], 

ISHM  may  be  one  answer  to  these  health  management  problems,  both  on  the  aircraft  and 
in  the  maintenance  and  logistics  side  of  operations.  Previous  research  efforts  have 
focused  solely  on  quantifying  the  cost  or  performance  benefits  provided  by  ISHM,  but 
few  have  looked  into  the  effect  of  ISHM  on  operational  effectiveness.  This  research 
effort  intends  to  give  decision  makers  a  better  understanding  of  the  advantages  and 
disadvantages  of  ISHM  by  adding  the  mission  environment  to  previously  built  cost  and 
performance  analyses. 

1.3  Research  Objectives  and  Hypothesis 

The  focus  of  this  research  is  to  quantify  the  mission-related  benefits  of  ISHM  by 
constructing  architecture  for  analysis  to  compare  against  current  autonomous  vehicle 
capabilities,  and  to  provide  a  general  baseline  model  that  can  be  implemented  over  any 
current  or  future  autonomous  vehicle.  The  architecture  will  include  the  ability  to  analyze 
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the  causal  relationship  of  ISHM  performance  metrics  (to  include  the  performance  of  the 
necessary  algorithms,  and  the  performance  and  reliability  of  the  monitoring  sensors)  to 
mission  performance.  The  research  presented  in  this  thesis  is  aimed  at  primarily 
answering  the  following  questions  using  the  architecture  in  a  modeling  and  simulation 
context: 

(1)  What  are  the  potential  impacts  to  ground  control  stations  and  users? 

(2)  What  are  the  potential  impacts  on  current  maintenance  practices? 

(3)  What  are  the  performance  characteristics  necessary  for  ISHM  to  effect 
mission  success? 

In  order  to  answer  these  research  questions  and  develop  an  appropriate  model,  a  literature 
review  should  first  be  conducted  to  answer  these  questions: 

(1)  What  is  the  current  status  of  UAS  health  management? 

(2)  What  are  the  essential  elements  of  ISHM? 

(3)  What  are  the  expected  benefits  of  ISHM? 

The  architecture  should  also  contribute  to  answering  these  secondary  questions: 

(1)  How  should  ISHM  data  be  presented  to  be  effective?  How  will  the 
presentation  change  in  regards  to  the  different  users  of  ISHM  (operators, 
maintenance,  etc.)? 

(2)  Is  ISHM  cost  effective? 


4 


1.4  Methodology 

The  development  of  the  analytic  architecture  simulating  ISHM  over  the  lifetime  of  the 
UAS  will  follow  the  eight- step  Architecture  Based  Evaluation  Process  and  be  in 
accordance  with  the  Department  of  Defense  Architecture  Framework  [DoD,  2012].  This 
process  was  developed  by  a  group  of  AFIT  graduate  students  in  2006  to  bridge  the  gap 
between  the  system  engineering  architecture  community  and  the  modeling  and  simulation 
community  [Dietrichs,  2006].  The  analytic  architecture  will  model  expected  mission 
success  rates  and  maintenance  actions  for  a  UAS,  both  with  and  without  ISHM  for 
statistical  comparison. 

1.5  Assumptions  and  Limitations 

In  order  to  make  a  general  baseline  model,  it  is  not  possible  to  represent  every  possible 
aspect  of  ISHM;  therefore,  there  are  several  limitations  to  this  research: 

(1)  This  architecture  does  not  analyze  ISHM  past  the  system  level;  however,  the 

architecture  can  be  easily  modified  to  include  components  and  subsystems 
that  are  of  value  to  the  researcher. 

(2)  At  this  point  in  ISHM  development,  the  ISHM  system  has  no  command  or 

control  over  the  autonomous  vehicle.  ISHM  in  its  current  technology  state 
only  provides  recommended  actions  based  on  the  type  of  fault  it  detects.  The 
vehicle’s  autonomous  management  system  will  prioritize  actions  as 
necessary  or,  if  there  is  time,  a  ground-based  ISHM  team  can  overrule  or 
direct  mitigation  actions  as  necessary. 
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1.6  Implications 

An  ideal  ISHM  would  have  several  major  benefits  over  current  maintenance  and  tasking 
practices.  By  having  a  real-time  autonomous  capability  to  detect,  isolate,  and  diagnose 
problems,  the  largest  direct  benefits  include  a  reduction  in  maintenance  time,  a  larger 
operational  flight  envelope,  and  the  ability  to  enable  e  collaborative  mission  re-planning 
based  on  current  system  capability  and  health.  Overall,  ISHM  would  improve  mission 
decision  making,  enable  condition-based  maintenance  and  provide  remaining  life 
quantification  while  reducing  current  conservative  design  life  margins  and/or  inspection 
intervals.  Planned  Near  Term,  Mid  Term,  and  Far  Term  future  capabilities  of  ISHM  are 
illustrated  in  Figure  1. 


is'fA 


Mission  Availability 

Condition-based  maintenance 
Remaining  life  quantification 
Misston-readinessdecision  making 
Improved  fault  isolation  and  detection 

Mission  Success 

Improved  systemsfault  detection, 
isolation,  and  accommodation 

Mission  Capability 
Maximized  vehicle  performance  based  on 
individual  response  to  environment 
Design  Paradigm 

Component  probabilistic  design 
Provide  QA  during  manufacture/use 
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Mission  Availability 

System  availability  feeding  mission  planning 
in  real  time 

Mission  Success 
Improved  systemsfault  detection,  Isolation, 
and  accommodation  -  system  wide 

Mission  Capability 
Maximized  vehicle  performance  based  on 
individual  response  to  environment 
Maximized  singlevehicle  capabilitiesbased 
on  complete  integration  of  systemshealth 
awareness 

Design  Paradigm 
Reduced  redundancy 
Reducing  conservative  design/life  margins 
Subsystem  probabilistic  design 


Mission  Availability 

Mission  Success 

Improved  systemsfault  detection,  isolation , 
and  accommodation 

Enabling  autonomous  mission  replanning 
based  on  actual  system  health  and 
capabilities 


Mission  Capability 

Enabling  collaborative  autonomous  mi 
replanning  based  on  actual  system 
health  and  capabilities 

Design  Paradigm 

System- wide  probabilistic  design 

Reduce/eliminate  certification  test 
requirements 

Enable  self-aware vehiclesto  define 
operating  envelope 


Near  Term  Mid  Term 

Figure  1  -  Future  ISFIM  Capabilities  [Derriso,  2011] 


Far  Term 
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1.7  Preview 


This  thesis  is  organized  into  five  chapters.  The  introductory  chapter  discusses  ISHM 
considerations  in  terms  of  technical  standards  and  through  the  system  architectural 
definition.  The  remaining  chapters  are  as  follows: 

•  Chapter  II  examines  and  classifies  the  current  state  of  health  management 
for  unmanned  aerial  systems,  provides  an  ISHM  system  taxonomy,  and 
summarizes  the  major  areas  of  research  currently  being  performed. 

•  Chapter  III  describes  the  research  methodology  and  introduces  the 
architectural  development  process  used  to  conduct  the  research. 

•  Chapter  IV  presents  a  proposed  prototypical  ISHM  architecture  and 
provides  analysis  of  the  analytic  architecture. 

•  Chapter  V  draws  conclusions  regarding  research  objectives,  answers  the 
investigative  questions  and  proposes  future  research. 
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II.  Literature  Review 


2.1  Chapter  Overview 

The  purpose  of  this  chapter  is  to  summarize  the  state  of  health  management  for 
unmanned  aerial  systems  (UAS),  provide  an  ISHM  system  taxonomy,  and  summarize  the 
major  areas  of  research  currently  being  performed.  The  first  section  discusses  current 
health  management  practices  and  describes  key  terminology.  The  second  section  provides 
a  description  of  a  typical  ISHM  system  as  described  by  literature  and  the  third  section 
lists  expected  benefits  and  applicable  metrics  of  ISHM.  The  fourth  section  describes  the 
main  modeling  approaches  for  analyzing  the  performance  or  cost-benefit  tradeoffs  of 
ISHM  systems.  The  fifth  section  gives  an  overview  of  current  analytic  architectures.  The 
last  section  summarizes  the  information  provided. 

2.2  Background 

In  order  to  quantify  the  effects  of  ISHM  on  a  system,  the  current  health  management 
practices  must  be  investigated  for  comparison;  this  section  discusses  current  health 
management  practices  and  describes  key  terminology. 

2.2.1  Current  State  of  UAS  Health  Management 

The  Air  Force  UAS  programs  currently  use  independent  sensors  incorporated  into  the 
vehicle’s  hardware  to  monitor  for  fault  indicators  on  critical  subsystems.  The  sensor  data 
is  continuously  transmitted  to  ground  operations  where  it  is  processed.  If  the  data 
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indicates  a  fault  has  occurred  the  ground  operator  executes  pre-determined  mitigation 


steps,  dependent  on  which  sensor  indicated  a  fault,  and  sends  a  message  to  maintenance. 
Once  the  vehicle  lands,  maintenance  personnel  perform  diagnostic  tests  to  confirm  the 
location  and  identify  the  type  of  fault,  and  then  perform  maintenance  actions  to  restore 
the  component.  This  is  less  a  health  management  system  than  a  health  monitoring  system, 
in  terms  of  nomenclature.  The  algorithms  used  for  these  systems  generally  only  indicate 
an  off-nominal  condition;  they  do  not  give  any  other  information  typical  of  a  health 
management  system. 

2.2.2  Taxonomy 

Understanding  the  ISHM  system  and  the  benefits  it  offers  depends  greatly  on 
understanding  several  key  terms:  failures,  errors,  faults,  novel  events,  fault  detection, 
fault  isolation,  diagnostics,  and  prognostics. 

Failure  is  defined  as  a  “deviation  in  behavior  between  the  system  and  its  requirements. 
Since  the  system  does  not  maintain  a  copy  of  its  requirements,  a  failure  is  not  observable 
by  the  system”  [Buede,  2000]. 

Error  is  defined  as  “a  subset  of  the  system  state,  which  may  lead  to  failure.  The  system 
can  monitor  its  own  state,  so  errors  are  observable  in  principle.  Failures  are  inferred  when 
errors  are  observed.  Since  a  system  is  usually  not  able  to  monitor  its  entire  state 
continuously,  not  all  errors  are  observable.  As  a  result,  not  all  failures  are  going  to  be 
detected  (inferred)”  [Buede,  2000]. 
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A  fault  is  a  known  “defect  in  the  system  that  can  cause  an  error.  Faults  can  be  permanent 
(e.g.,  a  failure  of  system  component  that  requires  replacement)  or  temporary  due  to  either 
an  internal  malfunction  or  external  transient.  Temporary  faults  may  not  cause  a 
sufficiently  noticeable  error  or  may  cause  a  permanent  fault  in  addition  to  a  temporary 
error”  [Buede,  2000]. 

A  novel  event  is  another  type  of  anomaly  in  the  same  class  as  a  fault.  The  difference  is 
that  a  fault  is  a  known  defect,  where  novel  events  are  unknown.  Prognostic  algorithms  are 
designed  to  respond  to  known  events  (faults),  not  novel  events  [Atlas,  2001]. 

Degradation  involves  a  declining  performance  measure  that  changes  with  time, 
particularly  to  a  lower  condition,  quality,  or  level.  Generally,  systems  will  continue  to 
operate  in  a  degraded  mode,  but  not  at  a  specified  operating  level.  Whether  the 
degradation  has  advanced  to  a  fault  or  failure  state  will  be  determined  as  part  of  the 
reliability  specification  [Ebeling,  2010]. 

Fault  detection  is  the  “determination  that  the  performance  of  a  system  or  subsystem  does 
not  correspond  to  its  expected  behavior.  In  more  general  terms,  it  is  determining  that  a 
failure  has  occurred”  [Ross,  1999]. 

Fault  isolation  is  the  “determination  of  the  specific  cause  of  failure  so  corrective  action 

can  be  taken. . .  Ideally,  systems  are  partitioned  such  that  measurable  functions  can  be 

implemented  on  the  lowest  repairable  assembly”  [Ross,  1999]. 
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Diagnostics  can  be  described  as  “the  process  of  locating  [a]  fault  at  the  level  in  which 
restoration  may  be  accomplished”  [Ebeling,  2010].  The  process  includes  the  utilization  of 
monitoring  hardware  and/or  software  to  detect  and  isolate  faults  in  a  given  system  [Clutz, 
2003].  In  some  expanded  definitions,  diagnostics  can  even  include  determination  of  a 
failure  cause  [Cardona,  1999].  For  the  purpose  of  this  thesis,  diagnostics  will  be  defined 
as  “the  utilization  of  monitoring  hardware  and/or  software  to  detennine  the  failure  cause 
by  detecting  and  isolating  faults.” 

Prognostics  is  defined  as  an  assessment  of  likely  future  health  (educated  prediction)  of  a 
piece  of  equipment,  based  on  current  information  [Cardona,  1999].  Prognostics  builds  on 
current  diagnostic  capabilities  using  automated  procedures  to  calculate  the  Estimated 
Time  to  Failure  of  a  system  or  component.  A  prognostics  system  is  often  associated  with 
condition-based  maintenance,  since  the  results  of  a  prognostic  analysis  indicates  required 
maintenance  actions,  either  real-time  or  predicted  [Clutz,  2003] . 

All  these  definitions  are  brought  together  in  Figure  2,  which  shows  a  typical  component 
health  trajectory.  Diagnostics  tells  “what”  fault  curve  the  component  is  on  and 
prognostics  detennines  “where”  on  the  overall  health  curve  the  component  currently 
resides. 
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Time,  t  Karr 

Figure  2  -  Component  Life:  100%  Healthy  to  100%  Failed  [Atlas,  2001] 

2.3  Notional  ISHM  Configuration 

An  ISHM  system  is  envisioned  to  serve  two  primary  goals:  to  monitor  the  “functional 
health”  of  the  system  real-time;  and  to  facilitate  the  maintenance  and  availability  of  the 
system  by  diagnosing  the  physical  break-downs  in  the  system  that  can  be  replaced  off¬ 
line.  These  two  goals  are  further  explained  below: 

1.  Real-time  monitoring  of  the  functional  health  of  the  system:  ISHM  must 
constantly  monitor  the  functional  health  of  the  system  to  detect  and  isolate 
faults.  From  this  standpoint,  the  system  is  regarded  as  a  ‘collection  of 
functional  units’  (rather  than  physical  units)  that  must  perform  flawlessly  to 
constitute  the  overall  function(s)  of  the  mission.  Depending  on  the  level  of 
autonomy,  criticality,  and  authority,  ISHM  could  either  make  ‘real-time’ 
decisions  to  reroute  flows  of  information,  energy,  or  material  from  the  failed 
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unit  to  ensure  continuous  operability,  or  send  appropriate  information  to  a 
human-in-the-loop  for  decision  making.  The  information  ISHM  provides  to  the 
decision  maker  should  have  integrity  and  be  relayed  within  enough  time  to 
facilitate  a  good  outcome. 

2.  Determining  the  physical  health  of  the  system:  To  help  maintain  the  physical 
health  of  the  system,  ISHM  must  be  able  to  determine  which  physical 
component  has  failed  or  is  going  to  fail  and  the  effect  of  the  failed  component 
on  the  system’s  capabilities.  By  continuously  monitoring  physical  units,  the 
information  collected  from  ISHM  should  also  be  used  to  identify  long-term 
degradation  effects  that  could  cause  failures  [Mehr,  2005]. 

Using  these  goals,  this  section  describes  a  notional  ISHM  configuration. 

A  typical  ISHM  system  consists  of  sensors  placed  at  critical  components  within 
subsystems  of  the  vehicle  that  stream  data  to  a  management  system.  The  management 
system  processes  the  sensor  data,  executes  diagnostic  and  prognostic  algorithms,  and  then 
feeds  this  information  through  a  reasoner,  as  seen  in  Figure  3.  This  management  system 
can  either  be  on-board  the  vehicle  in  a  hardware  configuration  or  off-board  enabling  the 
ground  command  and  control  (C2)  element. 
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Figure  3  -  Typical  ISHM  configuration  [Benedettini,  2009] 


Sensors  can  be  conventional,  measuring  temperature,  speed,  and  flow  rate,  or  specifically 
tailored  to  health  management  applications,  such  as  strain  gauges,  ultrasonic  sensors,  or 
proximity  devices.  The  sensor  data  is  then  processed  to  remove  any  artifacts  or  noises 
and  manipulated  to  extract  fault  features.  The  diagnostic  module  then  analyzes  the  fault 
features  to  detect,  identify,  and  isolate  developing  failure  conditions.  The  diagnostic 
information  will  be  combined  with  historical  data  in  the  prognostic  module  to  generate  an 
estimation  of  failure  times.  Algorithms  developed  for  the  diagnostic  and  prognostic 
modules  are  generally  based  on  mathematical  models  (e.g.,  Hamilton  dynamic, 
Lagrangian  dynamic,  approximation  methods),  or  pattern  recognition  (e.g.,  fuzzy-logic, 
statistical/regression  methods,  neural  network  clustering).  Finally,  the  diagnostic  and 
prognostic  information  is  turned  over  to  the  reasoner  module  which  analyzes  available 
resources,  decides  which  hazard  mitigation  steps  to  execute,  and  then  passes  the  selected 
decision  to  the  on-board  C2  module  and  relays  appropriate  information  to  the  ground  C2 
operator  and  maintenance  element  [Benedettini,  2009]. 
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2.4  Expected  Benefits  and  Applicable  Metrics 

The  overall  desired  effect  of  an  ISHM  system  would  be  to  continuously  monitor  the 
system,  detect  and  isolate  either  a  real-time  fault  or  pre-cursers  to  a  fault,  determine  the 
criticality  of  the  fault,  and  then  relay  appropriate  information  to  ground  control,  the  on¬ 
board  C2  module,  and  maintenance  for  action.  Benefits  of  this  capability  are  discussed  in 
the  remaining  sub-sections. 

2.4.1  Effect  on  Scheduled  and  Unscheduled  Maintenance 

The  Air  Force  goal  for  prognostic  systems  such  as  ISHM  is  to  completely  eliminate 
traditional  aircraft  inspection  and  repair  patterns  [Ross,  1999].  Currently,  a 
malfunctioning  unit  is  either  identified  in-flight  (based  off  an  alert  from  an  individual 
sensor)  or  identified  through  scheduled  inspections.  There  is  an  inherent  probability  of  a 
false  alarm  and  a  probability  of  fault  detection,  meaning  that  the  aircraft  could  be 
incorrectly  pulled  from  an  on-going  mission  or  could  continue  on  a  mission  with  an 
unknown  fault  that  could  lead  to  system  failure.  The  integrated  aspect  of  ISHM  proposes 
to  severely  reduce  the  false  alarm  rate  and  increase  the  total  probability  of  detection,  as 
understanding  the  full  health  status  of  the  vehicle  can  identify  false  positives  and  identify 
if  a  fault  or  failure  has  occurred  down-stream.  For  example:  a  sensor  falsely  identifies  a 
valve  stuck  closed,  a  sensor  further  down  the  stream  indicates  a  normal  flow  rate  and  the 
system  has  not  lost  any  performance  aspects;  ISHM  would  therefore  not  report  this  as  a 
system  fault,  but  as  a  sensor  fault. 
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With  the  continuous  monitoring  provided  by  ISHM,  pre-cursors  to  faults  can  also  be 
identified  and  Estimated  Time  to  Failures  of  the  component  or  total  system  will  be 
reported.  Additionally,  if  multiple  mission  data  is  stored,  every  time  a  fault  occurs,  the 
data  collected  by  ISHM  can  be  used  to  identify  new  indicators  or  pre-cursors  to  a  failure 
to  be  uploaded  into  the  diagnostic  and  prognostic  algorithms. 

The  overall  result  is  that  with  ISHM  implemented,  the  probability  of  unscheduled 
maintenance,  currently  inflated  due  to  prevalent  “Could  Not  Duplicate”  and  “No  Defect 
Found”  maintenance  results,  should  decrease  as  unscheduled  maintenance  should  become 
more  fault  driven.  Scheduled  maintenance  intervals  can  also  be  investigated  for  potential 
relaxation  or  removal;  current  intervals  may  be  conservatively  small  to  counteract  the 
current  lack  of  health  awareness.  Ideally  with  ISHM,  the  aircraft  would  replace  time- 
based  or  event-driven  maintenance  with  a  condition-based  maintenance  system,  where 
maintenance  is  only  performed  based  on  objective  evidence  of  actual  or  predictable 
failure  of  a  system  or  its  components  [OSAIDD,  1999]. 

Metrics  (unless  stated  otherwise,  all  formulas  in  this  Chapter  are  from  An  Introduction  to 
Reliability  and  Maintainability  Engineering  by  Charles  Ebeling,  2010): 

Tpm  -  Mean  Time  between  Performances  of  Preventative  (Scheduled)  Maintenance 
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MTBM  -  Mean  Time  between  Maintenance  (includes  both  scheduled  and  unscheduled 
maintenance).  The  equation  for  MTBM  is  shown  in  Equation  1. 

td 

MTBM  = - - — j—  (1) 

m(td)  +  jA- 
1  pm 


where  td  =  system  design  or  (economic)  life 

m(td)  =  expected  number  of  failures  in  the  interval  (0,  td) 

Ru  -  Rate  of  Unscheduled  Maintenance.  The  equation  for  Ru  is  shown  in  Equation  2. 

i(Tpm) 


Rn  - 


m[ 


(2) 


lpm 


where  m{Tpm)  =  expected  number  of  failures  in  the  interval  (0,  Tpm) 
MDT  -  Mean  Downtime.  The  equation  for  MDT  is  shown  in  Equation  3. 

i(td)MTTR  +  (t^-)  MPMT 


m( 


MDT  = 


(3) 


m(td)  +  j 


pm 


where  MTTR  =  Mean  Time  to  Repair 

MPMT  =  mean  preventative  maintenance  time 

Pd  -  Probability  of  Detection 

Pd  —  P (Fault  Indie ation\F ault  n  Operational  Sensor)  (4) 


Pfa  -  Probability  of  False  Alarm 

Pfa  —  P(Fault  Indie ation\N o  Fault  n  Operational  Sensor )  (5) 


2.4.2  Decreased  Mean  Time  to  Repair 

Current  fault  detection  is  limited  to  identifying  the  occurrence  of  a  fault  and  an 
approximate  location,  meaning  that  fault  isolation  can  only  occur  after  the  aircraft  has 
landed.  There  is  also  a  non-unity  probability  that  the  mechanic  can  even  correctly  identify 
the  failure  mode  once  it  lands.  With  ISHM,  both  fault  detection  and  isolation  are 
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performed  in-flight,  within  a  specified  confidence  level,  and  the  appropriate  information 
is  relayed  to  the  maintenance  element.  This  gives  the  maintenance  element  time  to  pre¬ 
position  the  necessary  maintenance  equipment  and  personnel  or  order  any  necessary 
replacement  parts,  severely  reducing  the  total  Mean  Time  to  Repair  (MTTR)  after  an 
event. 

As  a  result  of  its  continuous  monitoring,  ISHM  would  also  reduce  maintenance  time 
during  scheduled  inspections.  Prognostics  would,  theoretically,  calculate  an  Estimated 
Time  to  Failure  (ETF)  for  each  component,  resulting  in  each  inspection  only  focusing  on 
those  systems  that  had  passed  an  ETF  threshold  in  that  time  interval.  Knowing  that  the 
specific  systems  to  be  inspected  ahead  of  time  would  again  allow  the  maintenance 
element  time  to  pre-position  the  necessary  maintenance  equipment  and  personnel  or  order 
any  necessary  replacement  parts.  ISHM  would  also  negate  the  current  use  of  time¬ 
intensive  Built-in  Test  (BIT)  units,  as  each  system  would  be  continuously  tested. 

2.4.3  Operational  Availability  Improvement 

Based  on  the  decreased  downtime  in  unscheduled  maintenance  and  scheduled 

maintenance  from  ISHM,  the  Operational  Availability  for  each  aircraft  should  improve. 

Another  factor  affecting  Operational  Availability  is  mission  turn-around  time,  or  the  time 

from  when  the  aircraft  lands  to  when  it  is  ready  for  the  next  mission.  Without  ISHM, 

mission  turn-around  time  can  include  lengthy  BIT  tests  to  check  for  failures.  Since  these 

tests  are  not  needed  with  continuous  monitoring  and  a  higher  confidence  in  fault 

detection,  the  mission  turn-around  time  should  decrease,  increasing  Operational 
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Availability.  Whether  measured  in  maintenance  downtime  or  a  reduction  in  hours 
required  for  testing  and  diagnostics,  etc.,  the  net  result  is  that  a  system  with  ISHM  will  be 
available  for  use  more  of  the  time. 

Metrics: 

Ao  -  Operational  Availability.  The  equation  for  Aois  shown  in  Equation  6. 

MTBM 

A°  ~  MTBM  +  MDT  (6) 


2.4.4  Increased  Mission  Success 

Having  situational  awareness  of  the  entire  health  state  of  the  vehicle  assists  ground 
operations  in  providing  full  mission  coverage.  If  a  UAS  autonomously  detects  a  fault  and 
due  to  the  fault  criticality  (for  example,  low  fuel  levels)  decides  to  re-task  to  a  closer 
trajectory,  ground  operations  can  re-task  other  UAS  vehicles  to  ensure  coverage  of  the 
priority  targets.  Without  ISHM,  a  fault  alert  would  generally  give  no  indication  of  the 
remaining  performance  capability  of  the  UAS,  leaving  ground  operations  to 
conservatively  scrap  that  particular  mission  set. 

An  additional  aspect  of  increased  situational  awareness  is  its  affect  on  UAS  flight  limits. 
Modern  autonomous  flight  control  systems  limit  the  vehicle  to  safe  operating  loads  and 
environments;  this  operating  envelope  is  pre-defined  and  conservative.  With  ISHM,  the 
flight  envelope  can  theoretically  be  expanded  and  defined  by  the  design  criteria  for  the 
vehicle.  Health  data  would  then  be  used  to  restrict  the  envelope  to  a  prescribed  level  in 


19 


the  event  of  a  detected  fault.  This  would  increase  the  operational  capability  of  the  vehicle, 
allowing  for  larger  mission  sets.  Improved  situational  awareness  combined  with  the 
theoretical  improved  Operational  Availability  would  greatly  improve  the  rate  of  mission 
success. 

Metrics: 

Rms  -  Rate  of  Successful  Completed  Missions.  The  equation  for  RMs  is  shown  in 
Equation  7. 

#  Successful  Missions  ,nx 

Rm s  = - - -  (') 

Total  Missions  Attempted 


2.4.5  Cost  Savings 

The  previous  benefits  all  have  some  measure  of  cost  savings  attached  to  them.  Having  a 
lower  total  maintenance  downtime,  due  to  decreases  in  scheduled  maintenance  and  a 
lower  probability  of  unscheduled  maintenance,  leads  to  a  lower  personnel  cost  and  even 
an  option  of  having  less  maintenance  personnel  needed.  Fewer  maintenance  actions  also 
indicate  a  potential  reduction  in  spares  and  supply  costs.  However,  there  is  an  inherent 
cost  in  implementing  ISHM,  not  just  to  the  vehicle  but  to  the  resulting  operational 
infrastructure.  The  cost  savings  must  be  weighed  against  the  implementation  costs  to 
truly  investigate  the  financial  aspect  of  ISHM. 

Cost  avoidance  measures  could  also  be  applied  as  a  benefit  of  ISHM.  ISHM  identifies 
components  or  subsystems  that  are  near  failure,  replacing  or  repairing  these  parts  before 
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they  fail  and  cause  damage  to  other  parts  would  avoid  the  cost  of  repairing  the  additional 
damage.  The  upfront  cost  may  be  higher  in  the  short  run,  but  the  final  life  cycle  cost 
would  be  lower. 

2.5  Analytic  Models 

The  majority  of  analytic  models  for  ISHM  have  been  created  by  NASA  at  Ames 
Research  Center.  On-going  research  is  aimed  at  developing  a  robust  methodology  that 
can  evaluate  different  ISHM  architectures  to  optimize  a  set  of  pre-determined  metrics. 
This  process,  known  as  ISHM  Systems  Analysis  and  Optimization  (SA&O),  consists  of  a 
set  of  models  that  can  be  easily  customized  for  a  specific  system.  Using  this  SA&O 
process  offers  two  immediate  advantages: 

•  The  effects  of  ISHM  on  the  overall  safety,  maintainability,  and  performance  of 
the  system  can  be  calculated. 

•  During  design,  engineers  can  use  the  process  to  find  the  ‘optimal  ISHM 
architecture’  for  that  specific  system  [Mehr,  2005]. 

The  original  quantification  process  identified  24  metrics,  listed  in  Table  3,  to  be  used 
across  four  domains:  Design  for  Testability  (DFT)  Model,  Loss  of  Mission  (LOM) 
Model,  Turnaround  Model,  and  Maintenance  Model. 
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Table  3  -  ISHM  SA&O  Process  Metrics  [  Datta,  2004] 


1.  Loss  of  Mission 

13.  [UAS]/Subsystem  Mean  Time 
Between  Failure 

2.  Loss  of  Vehicle 

14.  Subsystem  Availability 

3.  Loss  of  Crew 

15.  [UAS]  Turnaround  Time 

4.  Launch  Availability 

16.  Cost  of  Spares 

5.  Development  Cost 

17.  [ISHM]  Weight 

6.  Production  Cost 

18.  Subsystem  Weight 

7.  Annual  Operational  Cost 

19.  Fault  Detection  Coverage 

8.  $/lb  (Mission  Price/lb) 

20.  Fault  Isolation  Coverage 

9.  Inherent  [ISHM]  Reliability 

21.  [ISHM]  False  Alarm  Rate 

10.  Subsystem  Reliability 

22.  Subsystem  False  Alarm  Rate 

11.  Subsystem  Failure 

Probability 

23.  Net  Present  Value  and  IRR  of 
[UAS]  program 

12.  [UAS]/Subsystem  Mean 

Time  To  Repair 

24.  Probability  of  unscheduled 
maintenance 

The  DFT  Model  assesses  the  ability  of  a  given  instrumentation  suite  to  detect  and  isolate 
the  faults  for  a  proposed  design,  the  size  of  ambiguity  groups,  and  test  point  selection; 
fault  detection  and  fault  isolation  metrics  are  derived  for  the  ISHM  system  from  the  DFT 
Model.  The  LOM  Model  assesses  the  probability  of  failures  that  result  in  an  inability  to 
complete  a  given  mission;  the  primary  output  for  the  LOM  Model  is  the  probability  of 
loss  of  mission  (as  a  metric).  The  Turnaround  Model  predicts  the  cost,  time  and  resources 
required  to  prepare  the  UAS  for  the  next  mission;  it  models  scheduled  and  unscheduled 
maintenance  and  the  repair  process.  Typically  this  model  uses  discrete  simulation  to 
output  the  new  (ideally,  lower)  UAS  turnaround  times  and  costs  of  operations.  The 
Maintenance  Model  is  used  to  provide  maintenance-related  input  on  a  subsystem-by- 
subsystem  basis  as  required  by  the  turnaround  and  mission  models.  Figure  4  maps  each 
of  the  metrics  to  each  other  and  to  the  relevant  model  [Datta,  2004] . 
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Figure  4  -  The  ISHM  SA&O  Quantification  Process  Map  [Datta,  2004] 

The  SA&O  process  was  found  to  have  several  shortcomings  that  hindered  its  application 
and  generalization  to  larger  and  more  complex  systems:  it  was  only  capable  of  producing 
a  ‘point-design’  instead  of  a  suite  of  design  alternatives,  and  it  did  not  take  into  account 
that  there  are  global  (shared)  as  well  as  local  design  parameters  for  each  subsystem. 
Building  from  the  SA&O  process  and  focusing  on  closing  these  gaps,  the  next  approach 
to  ISHM  analysis  is  known  as  ISHM  Multidisciplinary  Multi-objective  Systems  Analysis 
and  Optimization  (MMSA&O).  MMSA&O  structures  the  design  problem  into  a  two- 
level  hierarchical  architecture;  an  example  can  be  seen  in  Figure  5. 
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X-34:  Svstem-of- Systems 


Mm  F,  ,=IVHM  Price 
Min  Ft  ,= IYHM  Weight 
Max  Fi.i=  Maintainability 
Min  F<  ,=NPV  of  Costs 
Min  F;,i=  Dev  elopment  Cost 
Min  F„  ,=Prodncnon  Cost 
Min  Ft  =Operating  Cost 
Max  F,  =rVHM  Reliability 
Max  F1;=rVHM  Reliability 
Max  F,  n  t=IVHM  Rehability 

Exclusive  Objectives: 

Mm /=Fahe  Alarm  Rate 
Max/=Fault  Isolation 
Max/<=Fault  Detection 

Exclusive  Constraints 
fiJVHM  Design  C  onstraints 

Exclusive  Design  Variables. 
.Yi=(types  of  sensors.  HM 
reasoning  algorithms,  etc.) 


Performance: 

Mm  F  i=Mission  Price  lb 
Mm  F;= Vehicle  Weight 
Max  F  ,=Iaunch  Availability 
Costs: 

Min  Fi=Net  Present  Value  of  Costs 
Min  F  ^Development  Costs 
Min  F,=Producnon  Costs 
Min  F  ,=Annual  Operanag  Costs 
Risks: 

Min  F,=  Pr  (LOM) 

Min  F»=Pr  (LOC) 

Min  F,o=Pr  (LOV) 


Min  Fi  ,i=Vehicle  Price 
Min  F:  ,=  RLV  Weight 
Min  Fj.i=  RLV  Availability 
Min  F<  t=NPV  of  Costs 
Min  Fs  i-  Development  Cost 
Min  F„  i=Production  Cost 
Min  F-  =Ope:ating  Cost 
Min  F#  ,=  RLV  Reliability 
Min  Fv  i=  RLV  Reliability 
Min  Fio.i—  RLV  Reliability 

Exclusive  Objectiv  es: 

Min/,=MTTR 

Mm /=MTBF 

Min /i=Cost  of  Spares 

Min  /<= Pr  (Scheduled  Maintenance) 

Min /.=Pr  (Unscheduled  Maintenance) 

Min /,=Tm  around  time 

Exclusive  C  onstraints : 
g:: RLV  Design  Constraints 

Exclusive  Design  Variables: 

v.— (Physical  RLV  design,  components. 

RLV  architecture,  etc ) 


Remaining  Systems:  Crew  Vehicle  + 
ISS-Groimd  Station  (and  their  subsystems) 
lumped  into  one  discipline 


Min  Fi  _i=Price  of  utilizing  all  other  subsystems 


MinF; 
MinFv 
Min  F< 
Min  Fi 
Min  Ft, 
MinF, 
MinF, 
Mm  F, 


=Weight  of  other  attached  subsystems 
=Tum  Around  Time 
=NPV  of  Costs 
=  Development  Cost 
=Produchon  Cost 
=Operating  Cost 

=ReIiability  of  all  other  subsystems 
=Reliability  of  all  other  subsystems 


Min  Fi„  :=Re liability  of  all  other  subsystems 
Exclusive  Objectives: 

Min/,=Cost  of  Spares  (in  other  subsystems) 
Min /=Pr  (Scheduled  Maintenance) 

Min  /i=Pr  (Unscheduled  Maintenance) 

Min  /,=P r  (Scheduled  Maintenance) 

Exclusive  Constraints: 

gj  Design  constraints  for  all  other  subsystems 


Exclusive  Design  Variables: 

Xj=(Design  variables  in  all  other  subsystems) 


Figure  5  -  Example  Two-Tier  Formulation  of  an  ISHM  Design  Problem  for  a  Reusable  Launch 

Vehicle  (RLV)  [Mehr,  2005] 


In  this  process,  ISHM  is  decomposed  into  a  hierarchy  of  several  sub-problems,  each  of 
which  may  contain  multiple  objectives.  In  its  multi-disciplinary  form  (as  seen  in  Figure 
6),  the  optimization  problem  can  be  organized  into  two  levels:  one  sub-problem  at  the 
system  level,  and  J  sub-problems  at  the  sub-system  level. 
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System  level 


Sub-system  level 


Figure  6  -  Multi-Disciplinary  Form  of  a  Multi-Objective  Optimization  Problem  [Mehr,  2005] 

The  goal  of  this  optimization  approach  is  to  obtain  a  set  of  solutions  {xSH,  xlt ... ,  Xj)  that 
minimizes  a  weighted  sum  of  R  objectives  while  satisfying  the  constraints  in  all  J  sub¬ 
problems.  The  equivalent  single-level  form  of  the  multi-disciplinary  problem  is  seen  in 
Equation  8. 


minimize 

minimize 

fj(x±,Xj) 

subject  to : 

M-O  l 

[gj(x*-Xj)  J 

(8) 


where: 


Fjj  =  functionally-separable  objectives 

fj  =  exclusive  objectives 

xSH  =  shared  variable  vector 

Xj  =  variable  vector  exclusive  to  the  sub-system 

g0  =  system  constraint  vector 

g j  =  constraint  vector  exclusive  to  the  sub-system 
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The  solutions  from  each  sub-problem  are  then  rolled  up  to  the  top-level  for  integration; 


however,  since  each  sub-problem  is  solved  independently,  convergence  matrices  must  be 
used  to  guide  the  full  system  optimization  [Mehr,  2005]. 

The  SA&O  process  was  proven  to  significantly  improve  the  efficiency  of  ISHM 
architecture,  in  one  case  study  the  percentage  of  total  faults  detected  from  the  optimized 
ISHM  increased  to  75%  from  12%  in  the  original  design  [Mehr,  2005].  Likewise,  the 
improved  MMSA&O  has  seen  percentages  of  total  faults  detected  between  76  and  98% 
[Hoyle,  2007], 


Both  of  these  models  only  focus  on  the  safety,  maintainability,  and  performance  of  the 
new  ISHM-enabled  system.  These  models  are  missing  a  key  environment  that  is 
necessary  when  truly  evaluating  the  full  effect  of  ISHM:  the  mission  environment.  What 
is  the  effect  of  higher  availability  and  increased  situational  awareness  on  mission  success 
rates  over  the  lifetime  of  the  vehicle?  The  effect  on  mission  effectiveness  must  be 
quantified  to  help  fully  understand  the  cost/benefit  tradeoff  of  ISHM. 


2.6  Analytic  Architecture 

Historically,  architecture  and  modeling  had  been  performed  relatively  separately: 

“On  one  side  of  the  fence,  systems  engineers  . . .  [develop]  the  in- 
depth  integrated  architectures  to  define  system  concepts  for  development 
and  production.  On  the  other  side,  often  times  those  evaluating  the 
concepts  for  decision  makers  develop  simulations  and  models  from 
information  obtained  by  performing  their  own  research  and  interpretation 
of  the  system  concept.  The  result  of  this  disconnect  is  often  times  an 
inaccurate  evaluation  of  the  system  that  is  actually  developed  and 
produced’’  [Dietrichs,  2006]. 
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In  2006,  a  group  of  AFIT  graduate  students  bridged  this  gap  by  combining  the 
Department  of  Defense  Architecture  Framework  (DoDAF)  [DoD,  2012]  and  modeling 
techniques  into  an  analytic  architecture,  resulting  in  the  development  of  the  Architecture 
Based  Evaluation  Process  (ABEP)  [Dietrichs,  2006]. 


The  ABEP  is  made  up  of  the  following  eight  steps  (see  Appendix  A:  Architecture-Based 
Evaluation  Process  (ABEP)  for  the  process  assumptions  and  further  breakdown): 

1.  Design  Operations  Concept  of  System  to  be  evaluated. 

The  Ops  Concept  provides  the  system  operations  which  the  architecture  will 
model. 

2.  Identify  Measures  of  Effectiveness  (MOEs)  relevant  to  the 
decision/evaluation. 

Identify  the  mission  level  metrics  that  represent  the  effectiveness  of  the 
system. 

3.  Identify  required  level  of  abstraction  for  architecture  to  show  traceability 
to  MOEs. 

Analyze  the  Ops  Concept  to  determine  if  MOEs  are  measured  at  the  output  of 
a  system,  within  a  system,  or  at  the  output  of  activities  external  to  the  system. 

4.  Identify  architecture  views  necessary  to  capture  structure/relationships. 

a.  Structure  (OV-1,  OV-2,  and  OV-5  mandatory) 

b.  Decision  Logic  (OV-6a  mandatory) 

c.  As  Required:  SV-2,  SV-4,  SV-7,  OV-6b,  OV-6c 

5.  Develop  architecture  views. 

Develop  or  acquire  the  architecture  views  identified  in  Step  4  IAW  DoDAF  to 
include  all  relevant  activities  and  entities. 

6.  Develop  Modeling  and  Simulation  to  replicate  architecture. 

a.  Select  modeling  or  analytical  tools  best  suited  to  meet  evaluation 
requirements 

b.  Model  structure  of  simulation  or  analytical  solution  to  match 
architecture 

c.  Model  decision  logic  of  simulation  or  analytical  solution  to  match  OV- 
6a 

d.  Choose  input  parameters  consistent  with  SV  products 
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e.  Calculate  MOEs  at  output  of  activities  as  functions  of  design 
parameters 

7.  Evaluate  Model  Completeness. 

Determine  whether  model  considers  all  relevant  aspects  of  the 
system/concept. 

8.  Evaluate  model  for  MOE  results,  requirements,  and  key  parameters. 

a.  Once  the  model  is  complete,  evaluate  the  system’s  ability  to  meet 
target  metrics 

b.  Vary  design  parameters  and  perform  sensitivity  analysis  to  identify 
key  parameters 

c.  Compare  sensitivity  analysis  to  target  MOEs  to  help  establish/refine 
requirements  and  KPPs 

d.  If  not  already  accomplished,  develop  SV-7  Systems  Performance 
Parameters  Matrix  and  identify  critical  performance  parameters 

e.  Vary  system  design  and  design  parameters  to  evaluate  the  system’s 
robustness  and  its  rate  of  degradation 


2.7  Design  of  Experiments 

Design  of  Experiments  (DOE)  is  a  type  of  statistical  design  in  which 

“purposeful  changes  are  made  to  the  input  variables  of  a  process  or  system  so  that 
we  may  observe  and  identify  the  reasons  for  changes  that  may  be  observed  in  the 
output  response. . .  [These  experiments  are  planned]  so  that  appropriate  data  will 
be  collected  and  analyzed  by  statistical  methods,  resulting  in  valid  and  objective 
conclusions”  [Montgomery,  2009]. 

For  this  research  effort,  DOE  techniques  will  be  used  to  supplement  the  ABEP  when 
evaluating  models.  The  DOE  techniques  used  will  follow  the  seven  guidelines  provided 
in  Design  and  Analysis  of  Experiments  by  Dr.  Douglas  Montgomery  [2009]: 


1.  Recognition  of  and  Statement  of  the  Problem 

A  clear  statement  of  the  problem  provides  a  better  understanding  of  the  phenomenon 

being  studied  and  the  final  solution  of  the  problem.  It  is  important  to  keep  the  overall 

objective  in  mind  to  avoid  wasting  time,  materials,  and  other  resources. 
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2.  Selection  of  the  Response  Variable 

The  response  variable  or  variables  “provides  useful  information  about  the  process  under 
study.”  This  is  often  the  output  of  a  process,  or  a  measurable  characteristic  of  a  system. 
There  may  be  one  or  more  response  variables. 

3.  Choice  of  Factors,  Levels,  and  Range 

Design  factors  are  “those  [variables]  that  the  experimenter  may  wish  to  vary  in  the 
experiment.”  These  factors  are  expected  to  have  a  large  effect  on  the  response  variable. 
Once  the  experimenter  has  selected  the  factors,  they  “must  choose  the  ranges  over  which 
these  factors  will  be  varied  and  the  specific  levels  at  which  runs  will  be  made.”  A  very 
common  method  of  choosing  levels  is  to  select  a  high  and  a  low  point  that  covers  a  range 
that  the  experimenter  deems  is  appropriate  for  operating  conditions  or  is  of  interest  to  the 
experiment. 

4.  Choice  of  Experimental  Design 

Choosing  the  design  involves  consideration  of  sample  size,  selection  of  an  appropriate 
run  order,  and  determination  of  any  restrictions  in  the  design.  The  three  basic  principles 
of  experimental  design  are  randomization,  where  both  the  allocation  of  resources  and  the 
order  in  which  the  individual  trials  are  performed  are  randomly  determined;  replication, 
or  independent  repeats  of  each  factor  combination;  and  blocking,  a  design  technique  used 
to  improve  the  precision  with  this  comparisons  among  the  factors  of  interest  are  made. 
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A  common  experimental  design  that  combines  these  three  principles  is  a.  factorial 
experiment ,  in  which  factors  are  varied  together  instead  of  one  at  a  time.  This  particular 
experiment  enables  the  experimenter  to  easily  investigate  the  individual  effects  of  each 
factor  and  to  determine  where  the  factors  interact.  If  there  were  k  factors,  each  at  two 
levels  (high  and  low),  the  factorial  design  requires  2k  runs.  Generally  if  there  are  more 
than  five  factors,  it  becomes  cumbersome  to  run  all  possible  combinations  of  factor 
levels. 

Another  experimental  design,  a  fractional  factorial  experiment ,  is  a  variation  of  the 
factorial  experiment  in  which  only  a  subset  of  the  runs  are  used.  These  designs  rely  on 
the  experimenter  assuming  that  certain  high-order  interactions  are  negligible,  and  that  the 
important  information  is  found  in  the  main  factors  and  low-order  interactions.  This  is  also 
known  as  the  sparsity  of  effects  principle.  A  major  use  of  this  experiment  is  for  screening 
factors  to  identify  those  factors  (if  any)  that  have  large  effects  on  the  response. 

5.  Performing  the  Experiment 

It  is  vital  to  monitor  the  process  carefully  to  ensure  that  the  procedure  is  executed 
according  to  plan.  Errors  in  procedure  will  usually  destroy  experimental  validity. 

6.  Statistical  Analysis  of  the  Data 

If  the  experiment  has  been  designed  correctly  and  performed  according  to  the  design,  the 

statistical  methods  required  can  be  simple.  The  output  of  the  experiment  should  be  a 

model  that  describes  the  response  surface  of  the  process  or  system  being  investigated. 
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The  most  commonly  used  statistical  inference  procedure  to  validate  this  model  is  the 
Analysis  of  Variance,  which  relies  on  portioning  the  total  variability  into  its  component 
parts:  variance  due  to  the  model,  and  variance  due  to  random  error.  Certain  assumptions 
have  to  be  satisfied  for  this  procedure  to  be  implemented,  specifically  that  the  errors  are 
normally  and  independently  distributed  with  mean  zero  and  constant  but  unknown 
variance  a".  Violations  of  these  assumptions  can  be  investigated  by  examination  of  the 
residuals,  or  the  difference  between  the  observed  value  and  the  predicted  value.  If  the 
model  is  valid  the  residuals  should  be  structureless,  or  containing  no  obvious  patterns. 

7.  Conclusions  and  Recommendations 

Once  the  data  has  been  statistically  analyzed,  the  experimenter  can  draw  practical 
conclusions  about  the  results  and  recommend  a  course  of  action.  Confirmation  testing  can 
be  performed  to  validate  the  conclusions,  if  necessary.  Often,  full  investigation  of  the 
response  surface  involves  iterative  experimentation,  as  each  new  experiment  builds  on 
the  conclusions  found  in  the  last. 

2.8  Literature  Review  Summary 

This  section  discussed  and  identified  several  key  terms,  such  as  faults,  failures, 
prognostics  and  diagnostics,  that  are  necessary  for  understanding  an  ISHM  system  and 
identified  current  health  management  practices  for  Unmanned  Aerial  Systems.  A  typical 
ISHM  configuration  was  introduced  and  had  the  following  components:  a  sensor  suite 
placed  along  critical  system  elements,  and  a  management  component  that  included  sensor 
data  processing,  diagnostic  and  prognostic  algorithms  to  identify  current  or  incipient 
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faults,  and  a  reasoner  to  select  the  appropriate  mitigation  steps  to  execute.  The  expected 
performance,  maintenance,  and  mission  benefits  of  adding  a  typical  ISHM  configuration 
to  a  UAS  were  identified  and  discussed. 

Prior  analytic  models  were  also  investigated.  Most  published  research  concerning 
analytic  modeling  of  Integrated  System  Health  Management  were  found  to  be  generally 
concerned  with  quantifying  the  effects  of  ISHM  on  the  performance  and  scheduled 
maintenance  of  the  intended  recipient  system.  Few,  if  any,  addressed  the  effect  of  ISHM 
on  mission  success  rates;  most  that  did  addressed  this  aspect  at  a  mission  level  and  did 
not  address  the  system  degradation  that  would  occur  over  the  lifetime  of  the  vehicle. 
Finally,  this  chapter  discussed  the  analytic  architecture  process  model  that  will  be  used  in 
Chapter  III  to  help  quantify  the  effect  of  ISHM  on  mission  success  rates. 
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III.  Methodology 


3.1  Overview 

The  purpose  of  this  chapter  is  to  describe  the  process  of  developing  an  analytic 
architecture  to  be  used  to  evaluate  the  advantages  and  disadvantages  of  installing  an 
ISHM  system  on  a  UAS.  The  development  will  follow  the  eight-step  Architecture  Based 
Evaluation  Process  (ABEP)  described  in  Section  2.6,  and  will  be  IAW  the  Department  of 
Defense  Architecture  Framework  (DoDAF). 

3.2  Design  ISHM  Concept  of  Operations 

Per  Step  1  of  the  ABEP,  Concept  of  Operations  (CONOPs)  will  be  developed  based  upon 
discussions  with  the  users.  To  help  organize  the  competing  objectives  of  ISHM  and 
ISHM’s  analysis,  two  CONOPs  will  be  built:  the  first  detailing  the  ISHM  system  to  be 
implemented,  the  second  focusing  on  the  analytic  architecture  model.  The  CONOPs  will 
adhere  to  Air  Force  Policy  Directive  10-28  and  will  outline  basic  Measures  of 
Effectiveness  (MOEs),  sequences  of  events,  command  relationships,  and  the  expected 
output  data  from  the  model. 

The  ISHM  CONOPs  is  meant  to  be  as  general  as  possible  and  will  take  a  system-level 
view  of  the  technology.  Capabilities  and  characteristics  will  be  taken  mostly  from  the 
research  completed  in  the  literature  review  with  implementation  directed  at  an  Unmanned 
Aerial  System. 
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The  purpose  of  the  Analytic  Architecture  CONOPs  is  to  primarily  answer  the  research 
questions  posed  in  Chapter  I.  For  the  purpose  of  this  research  effort,  the  architecture 
created  will  provide  a  general  baseline  model  that  can  be  implemented  over  any 
autonomous  vehicle.  The  architecture  will  be  built  using  the  characteristics  and 
capabilities  detailed  in  the  ISHM  CONOPs  and  will  be  used  to  design  an  analytic  model 
that  quantifies  the  effect  of  ISHM  on  the  operational  availability  and  mission  success 
rate. 

3.3  Identify  Measures  of  Effectiveness 

The  next  two  steps.  Step  2  and  3,  continue  development  with  the  creation  and  analysis  of 
a  list  of  MOEs  to  be  used  to  evaluate  ISHM.  MOEs  should  primarily  be  derived  from  the 
expected  benefits  of  ISHM.  Section  2.4,  Table  3  and  the  ISHM  CONOPs  built  in  the 
previous  section  list  several  metrics  that  have  already  been  identified  as  pertaining  to  the 
performance  of  ISHM.  The  MOEs  chosen  should  reflect  the  purpose  and  desired  output 
of  the  analytic  model,  as  they  will  ultimately  guide  the  development  of  the  model; 
leaving  out  key  evaluation  metrics  would  cause  an  inappropriate  output  from  the  model. 

Once  the  MOEs  are  chosen,  they  should  be  analyzed  against  the  CONOPs  to  determine 
where  in  the  system  (within,  at  the  output,  through  an  external  system)  they  are 
measured.  The  MOEs  will  also  be  used  to  identify  within  the  overall  system’s 
architecture  those  products  that  specifically  addressed  ISHM.  From  these  products,  a 
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Rules  Model  can  be  built  that  will  abstract  activities  and  will  serve  as  the  basis  for  the 
ISHM  simulation. 

For  this  research  effort,  the  analytic  architecture  will  have  the  capability  to  ingest  system 
failure  characteristics,  in  this  case  an  appropriate  failure  distribution  that  models  the  total 
system  as  well  as  probabilities  of  occurrence  for  the  fault  categories  listed  in  Table  4,  and 
ISHM  performance  characteristics,  such  as  the  probability  of  detection,  the  probability  of 
a  false  alarm,  and  the  diagnostic  algorithm  confidence  level  (a  probability  that  the 
diagnostic  subsystem  will  correctly  identify  the  fault).  These  categories  are  not  exclusive 
to  degradation  effects;  the  Estimated  Time  to  Failure  could  be  calculated  for  a  component 
experiencing  long-term  system  degradation  due  to  normal  wear  and  tear  or  for  a 
component  operating  at  a  high  level  of  stress.  The  analytic  architecture  will  then  have  the 
capability  to  use  those  input  variables  to  calculate  these  metrics  for  a  UAS  with  ISHM 
and  without  ISHM  for  comparison:  number  of  unscheduled  maintenance  actions,  and  the 
rate  of  mission  success. 
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Table  4  -  Fault  Categories 


Fault  Category 

Category  Definition 

I 

The  calculated  Expected  Time  to  Failure  is  much 
greater  than  mission  length.  Maintenance  can  wait 
until  the  next  scheduled  Preventative  Maintenance 
activity. 

II 

The  calculated  Expected  Time  to  Failure  is  greater 
than  mission  length.  Unscheduled  Maintenance 
must  occur  after  the  current  mission  is  completed. 

III 

The  calculated  Expected  Time  to  Failure  is  less 
than  mission  length,  but  mission  can  still  be 
completed  with  reduced  capability.  Unscheduled 
Maintenance  must  occur  after  the  current  mission  is 
completed. 

IV 

The  calculated  Expected  Time  to  Failure  is  less 
than  mission  length  and  the  UAS  must  abort  the 
mission  and  return  to  base  immediately. 

Unscheduled  Maintenance  must  occur  as  soon  as 
possible. 

V 

Catastrophic  Damage  expected  from  fault.  Loss  of 
vehicle  occurs 

3.4  Identify  and  Develop  Architecture  Views 

Step  4  and  5  identifies  and  then  develops  the  architecture  views  necessary  to  capture  all 
the  inter-relationships.  The  ABEP  offers  several  mandatory  and  recommended  products 
that  should  be  developed  to  cover  the  overall  structure  and  decision  logic.  Using  the 
previously  developed  CONOPs  as  the  basis,  nearly  all  evaluations  will  require  an  OV-1 
(High  Level  Operations  Concept)  and  OV-2  (Operational  Node  Connectivity 
Description),  and  all  will  require  an  OV-5b  (Operational  Activity  Model).  The  level  of 
abstraction  for  the  OV-5  will  have  been  identified  in  the  previous  section.  For  the 
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decision  logic,  an  OV-6a  (Rules  Model)  will  be  developed  to  match  the  level  of 
abstraction  used  for  the  OV-5. 

Additional  necessary  views  will  be  identified  through  the  CONOPs  and  the  selected 
MOEs.  Some  additional  views  called  out  by  the  ABEP  that  have  been  used  in  the  past 
include  the  SV-2  (Systems  Resource  Flow  Description),  SV-4  (Systems  Functionality 
Description),  SV-7  (Systems  Measures  Matrix),  OV-6b  (State  Transition  Diagram),  and 
OV-6c  (Event-Trace  Description).  All  identified  views  will  then  be  developed  IAW 
DoDAF  guidelines. 

Current  views  planned  for  this  research  effort  are  displayed  in  Table  5  along  with  their 
purpose. 
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Table  5  -  Planned  Architecture  Views 


Operational  Views 

Purpose 

OV-1  High  Level  Operations 

Concept 

Provides  a  graphical  depiction  of  what  the 
architecture  is  about  and  an  idea  of  the 
players  and  operations  involved 

OV-2  Operational  Node 

Connectivity  Description 

Depicts  Operational  Needlines  (flows  of  funding, 
personnel  and  materiel  in  addition  to 
information)  that  indicate  a  need  to  exchange 
resources 

OV-5a  Operational  Activity 
Decomposition  Tree 

Decomposes  the  operational  activities  that  are 
normally  conducted  in  the  course  of  achieving  a 
mission 

OV-5b  Operational  Activity 

Model 

Describes  input/output  flows,  dependencies  and 
relationships,  and  external  interchanges  between 
operational  activities 

OV-6a  Rules  Model 

Describes  the  rules  under  which  the  architecture 
behave  under  specified  conditions 

System  Views 

Purpose 

SV-1  Systems  Interface  Model 

Depicts  all  System  Resource  Flows  between 
Systems  that  are  of  interest 

All  Views 

Purpose 

AV-1  Overview  and  Summary 
Information 

Provides  executive-level  summary  information  in 
a  consistent  form  that  allows  quick  reference  and 
comparison  between  views. 

3.5  Develop  Analytic  Modeling  and  Simulation 

Selecting  the  modeling  or  analytical  tools  best  suited  to  meet  the  purpose  of  the  analysis 
is  Step  6  of  the  ABEP.  The  model  should  be  consistent  with  the  architecture:  the  structure 
should  match  the  OV-2  and  OV-5b  products,  the  decision  logic  should  be  based  off  of  the 
OV-6a,  and  the  parameters  should  be  consistent  with  the  systems  described  in  the  SV-1. 
The  additional  views  will  not  be  directly  involved  with  the  analytic  model  but  are 
required  to  ensure  the  architecture  products  are  consistent  with  each  other:  the  OV-1  and 
CV-1  provide  general  overviews  for  each  viewpoint,  the  SV-5b  ensures  that  the 
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operational  activities  in  the  OV’s  are  matched  to  the  ISHM  systems  described  in  the  SV- 
1,  and  the  AV-1  ties  all  the  views  together. 

For  this  evaluation  of  the  effect  of  ISHM  on  mission  effectiveness,  a  spreadsheet  model 
will  be  built  in  Microsoft  Excel.  The  model  will  run  over  the  lifetime  of  a  UAS,  whose 
failure  characteristics  serve  as  an  input  to  the  model,  and  will  output  unscheduled 
maintenance  actions  and  mission  success  rates  using  ISHM  performance  characteristics. 
Each  lifetime  will  be  considered  a  Monte  Carlo  event,  with  each  scheduled  maintenance 
interval  or  unscheduled  maintenance  repair  acting  as  a  renewal  process  for  the  UAS  (a 
process  that  restores  the  vehicle  to  “its  original  or  ‘as  good  as  new’  condition”)  [Ebeling, 
2010], 

The  full  list  of  parameters  needed  for  the  model  and  their  definitions  are  displayed  in 
Table  6;  the  inputs  are  divided  between  characteristics  of  the  UAS  and  performance 
measures  of  the  proposed  ISHM  addition,  the  outputs  are  divided  between  expected 
maintenance  actions  and  a  calculated  rate  of  mission  success  as  defined  by  Equation  7. 
The  user  can  also  select  how  many  Monte  Carlo  simulations  to  execute,  up  to  500 
iterations  of  the  lifetime  of  the  UAS. 
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Table  6  -  ISHM  Analytic  Model  Parameters 


UAS  Properties 
(Input) 

Definitions 

P(Failure) 

Probability  of  a  failure  occurring;  a  failure  distribution 

P(Fault  Categories) 

If  a  fault  occurs,  the  probability  of  it  falling  into  each  of  the  five 
fault  categories;  a  number  between  0  and  1  for  each  fault  category 

Average  Mission 
Length 

The  average  mission  length  for  the  UAS;  in  hours 

Scheduled 

Maintenance  Interval 

The  interval  between  scheduled  maintenance;  in  hours 

Expected  System 
Lifetime 

The  expected  lifetime  of  the  UAS;  in  hours 

ISHM  Properties 
(Input) 

Definitions 

PD 

Probability  of  detecting  a  fault;  between  0  and  1 

P (F ault  Indie ation\F ault  n  Operational  Sensor) 

PPA 

Probability  of  the  sensor  reading  a  false  alarm;  between  0  and  1 
P(Fault  Indie ation\N o  Fault  n  Operational  Sensor) 

Dcl 

ISHM’s  Diagnostic  Confidence  Level,  or  the  strength  of  the 
prognostic  and  diagnostic  algorithms;  between  0  and  1 

Expected  Model 
Output 

Definitions 

Baseline 

Maintenance  Actions 

Expected  number  of  maintenance  actions  for  a  UAS  using  current 
health  management  practices 

Baseline  Rate  of 
Mission  Success 

Expected  rate  of  mission  success  for  a  UAS  using  current  health 
management  practices 

ISHM  Maintenance 
Actions 

Expected  number  of  maintenance  actions  for  the  baseline  UAS 
with  the  addition  of  ISHM 

ISHM  Rate  of 
Mission  Success 

Expected  rate  of  mission  success  for  the  baseline  UAS  with  the 
addition  of  ISHM 

Model  Properties 

Definitions 

Number  of 
Simulations 

Number  of  lifetime  simulations  to  execute;  from  1  to  500 

3.6  Evaluate  Model 

The  model  will  then  be  evaluated  for  completeness  and  ability  to  meet  the  target  metrics 
in  the  final  Steps  7  and  8.  In  Step  7,  the  model  is  evaluated  solely  on  its  ability  to 
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consider  all  relevant  aspects  (processes,  assumptions,  input  variables,  output  data  and 
MOEs)  of  the  concept.  If  the  model  is  determined  to  not  be  complete,  the  process  will 
return  to  Step  3  with  some  additional  considerations  (listed  in  Appendix  A:  Architecture- 
Based  Evaluation  Process  (ABEP)).  If  the  model  is  considered  complete,  the  process  will 
proceed  to  Step  8. 


The  final  step  deals  with  the  results  of  the  model.  Representative  data  for  a  UAS  will  be 
fed  into  the  model  and  Design  of  Experiments  (DOE)  techniques  will  be  used  to 
determine  situations  where  ISHM  can  be  effectively  used.  The  response  for  this  analysis 
is  the  difference  between  the  number  of  successful  missions  calculated  for  a  system 
without  ISHM  (i.e.  using  current  health  management  techniques)  and  a  system  with 
ISHM.  The  intent  is  to  explore  the  response  surface  where  this  difference  is  maximized, 
which  coincides  with  the  operational  area  where  ISHM  would  be  most  beneficial. 
Representative  data  can  be  found  in  Table  7. 


Table  7  -  Representative  UAS  Data 


UAS  Properties 

Values 

P(Failure)  -  Distribution 

Weibull  Distribution 

Average  Mission  Length 

10  hours 

Scheduled  Maintenance  Interval 

1 ,000  hours 

Expected  System  Lifetime 

10,000  hours 

Without  actual  UAS  failure  data,  a  Weibull  failure  distribution  for  P(Failure)  was  chosen 
because  of  its  ability  to  model  the  minimum  of  a  large  number  of  independent  positive 
random  variables  from  several  classes  of  distributions  (i.e.,  the  distribution  is  great  at 
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modeling  a  system  of  systems  where  a  failure  in  one  component  causes  a  system-level 
failure)  [Meeker,  1998].  The  scale  and  shape  parameters  will  be  left  up  to  DOE  analysis 
to  determine  the  region  where  the  response  is  maximized. 

The  average  mission  length,  scheduled  maintenance  interval,  and  expected  lifetime  were 
chosen  by  the  researcher  to  represent  a  typical  UAS.  They  do  not  reflect  any  specific 
aircraft  in  the  USAF  inventory. 

3.7  Summary 

This  section  went  into  detail  as  to  how  the  ABEP  is  used  to  create  an  analytic  model  for 
the  purpose  of  evaluating  ISHM.  The  architecture  that  will  be  built  for  the  purposes  of 
this  research  effort  will  represent  a  general  ISHM,  as  researched  in  Chapter  II.  The 
analytic  model  based  off  this  architecture  will  be  focused  primarily  on  analyzing  mission 
effectiveness  using  generated  unscheduled  maintenance  actions  and  mission  success 
rates.  The  architecture,  model,  and  model  results  are  described  in  detail  in  Chapter  IV. 
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IV.  Analysis  and  Results 


4.1  Chapter  Overview 

This  chapter  will  present  the  completed  architecture,  resulting  analytic  model,  and  an 
analysis  of  representative  UAS  failure  data. 

4.2  ISHM  Architecture 

This  section  details  the  architecture  developed  using  the  methodology  in  Chapter  III. 
Since  the  focus  of  this  research  effort  is  on  the  analytic  nature  of  the  architecture,  only 
views  directly  relevant  to  the  analytic  model  will  be  discussed  in  detail  in  this  section. 
The  full  system  architecture  can  be  found  in  Appendix  B:  ISHM  Architecture. 

4.2.1  Integrated  Systems  Health  Management  Concept  of  Operations 

The  architecture  relies  heavily  on  a  robust  concept  of  operations,  especially  when 
designing  the  systems  and  operations  viewpoints.  The  full  concept  of  operations  for  a 
typical  ISHM  system  can  be  found  in  Appendix  B.l  Integrated  System  Health 
Management  Concept  of  Operations,  but  for  the  purposes  of  understanding  the  resulting 
viewpoints  in  this  chapter,  critical  portions  of  the  necessary  capabilities,  enabling 
capabilities,  sequenced  actions,  and  command  relationships  are  described  below. 
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Necessary  Capabilities  -  Data  Management 

The  ISHM  system  must  provide  continuous  monitoring  over  the  entirety  of  the  vehicle. 
Sensors  are  placed  in  critical  locations  in  order  to  feed  information  on  the  state  of  the 
system.  Sensors  can  be  conventional,  measuring  temperature,  speed,  and  flow  rate,  or 
specifically  tailored  to  health  management  applications,  such  as  strain  gauges,  ultrasonic 
sensors,  or  proximity  devices. 

Data  Management  also  includes  parameter  sets,  vehicle  configuration,  and  a  data  store 
with  a  list  of  safe  states  associated  with  known  fault  events  and  mitigation  steps.  Current 
mission  sensor  data  and  event  recording  can  either  be  kept  in  an  on-board  data  storage 
system  sent  to  ground  as  required,  or  continuously  streamed  to  ground  control. 

Necessary  Capabilities  -  Fault  Detection 

The  sensor  data  is  then  processed  to  remove  any  artifacts  or  noises  and  manipulated  to 
extract  fault  features  (either  current  or  pre-cursers)  and  provide  a  comprehensive  system 
picture.  Fault  Detection  combines  diagnostic  information  with  historical  data  (prognostic 
reasoning)  to  generate  an  estimation  of  failure  times.  These  fault  indications  are  then 
sorted,  prioritized,  and  distributed  to  insure  action  within  time  to  criticality.  Algorithms 
developed  for  diagnostic  and  prognostic  calculations  are  generally  based  on  mathematical 
models  (e.g.  Hamilton  dynamic,  Lagrangian  dynamic,  approximation  methods),  or 
pattern  recognition  (e.g.  fuzzy-logic,  statistical/regression  methods,  neural  network 
clustering). 
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Necessary  Capabilities  -  Fault  Isolation 

After  identifying  that  a  fault  has  occurred,  ISHM  must  pinpoint  the  fault  mechanism  (i.e. 
the  specific  cause  of  failure)  and  its  location.  If  not  identifiable  through  prognostic  or 
diagnostic  reasoning,  common  fault  mechanisms  for  that  location  can  be  identified  using 
historical  failure  data. 

Necessary  Capabilities  -  Health  State  Assessment 

ISHM  must  have  the  capability  to  assess  and  assign  levels  of  health  to  the  vehicle.  This  is 
achieved  by  calculating  the  remaining  vehicle  capabilities  based  on  a  capability  model 
and  the  current  fault  state  of  the  system.  A  notional  capability  model  is  hierarchically 
based,  where  the  higher-level  capability  is  computed  using  the  values  of  the  lower-level 
capabilities  and  a  mathematical  expression.  Faults  are  quantified  at  the  lowest  level  with 
system-level  capability  computations  that  orient  this  data  with  mission  requirements  to 
determine  effects  on  the  vehicle. 

Necessary  Capabilities  -  Select  Mitigation  Procedures 

The  ISHM  system  will  provide  mitigation  procedures  in  the  event  of  a  known  fault  for 

the  on-board  flight  control  to  act  on  if  necessary.  In  order  to  perform  this  capability, 

ISHM  will  a)  examine  the  available  resources  to  determine  any  performance  limitations 

and  to  estimate  the  time  to  criticality;  b)  confirm  the  diagnosed  event  and  declare  it  to  be 

a  valid  vehicle  event  with  a  high  confidence  level;  c)  access  the  fault  data  store  for  the 

appropriate  safe  state  and  the  feasible  step  alternatives;  before  d)  selecting  the  action 

steps  that  allow  completion  within  the  criticality  time  and  performance  limitations.  These 
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action  procedures  will  then  be  sent  to  the  on-board  flight  control  and  to  ground  control. 
Since  ISHM  operates  only  on  known  faults  and  known  mitigations,  any  unknown  fault 
will  immediately  be  assigned  a  critical  level  of  health  and  the  aircraft  will  automatically 
return  to  base. 

Enabling  Capabilities 

A  formal  Failure  Mode,  Effect,  and  Criticality  Analysis  (FMECA)  must  be  performed  on 
the  vehicle  prior  to  ISHM  being  implemented  [Ebeling,  2010].  This  is  an  iterative  process 
that  identifies  failure  modes,  assesses  their  probabilities  of  occurrence  and  their  effects  on 
the  system,  isolating  their  causes,  and  determining  corrective  action  or  preventative 
measures.  The  results  of  the  FMECA  should  identify  critical  sub-systems  or  components 
where  sensors  need  to  be  applied,  guide  the  diagnostic  and  prognostic  algorithm  creation, 
and  assign  criticality  to  failure  modes  for  health  assessment  purposes. 

Sequenced  Actions  -  Nominal  Operations 

The  ISHM  system  will  be  continuously  monitoring  the  health  state  of  the  UAS  and  will 
communicate  either  continuously  or  on  set  intervals  (barring  a  fault  event)  the  health 
status  of  the  UAS.  ISHM  will  also  be  continuously  calculating  an  Estimated  Time  to 
Failures  for  every  monitored  component. 
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Sequenced  Actions  -  Real-Time  Fault  Event 

Once  a  failure  occurs,  the  following  actions  should  take  place: 

(1)  ISHM  locates  the  fault  and  identifies  the  failure  mode 

(2)  ISHM  assigns  a  criticality  to  the  fault  mode  and  adjusts  the  vehicle’s  health 
status  to  the  appropriate  level 

(3)  ISHM  evaluates  the  new  capability  of  the  vehicle 

(4)  ISHM  selects  appropriate,  deterministic,  mitigation  action  procedures, 
correlating  them  with  mission  and  vehicle  inhibits. 

(5)  ISHM  sends  the  action  procedures  to  the  on-board  flight  control,  and  alerts  the 
Ground  Operator  and  the  Maintenance  element 

a.  The  on-board  flight  control  can: 

(i)  Continue  on  current  trajectory  (ignore  ISHM) 

(ii)  Use  recommendations  to  autonomously  reconfigure  and/or 
reshape  the  current  trajectory 

b.  The  Ground  Operator,  as  appropriate  and  in  accordance  with  the 
criticality  of  the  event,  can: 

(i)  Override  the  on-board  flight  control  decision  and  re-task  within 
its  new  capability 

(ii)  Defer  to  the  autonomous  on-board  flight  control  decision 

c.  The  Maintenance  element  executes  maintenance  actions  as  appropriate 
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Sequenced  Actions  -  Pre-Cursor  to  Fault  is  Detected 

When  a  pre-cursor  to  a  fault  is  detected,  the  following  actions  should  take  place: 

(1)  ISHM  locates  the  affected  component  and  identifies  the  impending  failure 
mode 

(2)  ISHM  calculates  an  Estimated  Time  to  System  (or  Component)  Failure 

(3)  ISHM  assigns  a  criticality  to  the  fault  mode  and  adjusts  the  vehicle’s  health 
status  to  the  appropriate  level 

(4)  ISHM  evaluates  the  new  capability  of  the  vehicle 

(5)  ISHM  selects  appropriate,  deterministic,  mitigation  action  procedures, 
correlating  them  with  mission  and  vehicle  inhibits. 

(6)  ISHM  sends  the  action  procedures  to  the  on-board  flight  control,  and  alerts  the 
Ground  Operator  and  the  Maintenance  element 

a.  The  on-board  flight  control  can: 

(i)  Continue  on  current  trajectory  (ignore  ISHM) 

(ii)  Use  recommendations  to  autonomously  reconfigure  and/or 
reshape  the  current  trajectory 

b.  The  Ground  Operator,  as  appropriate  and  in  accordance  with  the 
criticality  of  the  event,  can: 

(i)  Override  the  on-board  flight  control  decision  and  re-task  within 
its  new  capability 

(ii)  Defer  to  the  autonomous  on-board  flight  control  decision 

c.  The  Maintenance  element  executes  maintenance  actions  as  appropriate 
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Command  Relationships  -  Ground  Control 

Ground  systems  are  normally  treated  as  separate  systems,  and  their  relationship  to  the 
vehicle  has  typically  been  one  of  controller  and  operator;  in  this  case,  ground  is 
hierarchically  superior  to  the  vehicle  and  commands  it  for  some  mission  phases  but  is 
reactionary  for  others.  Vehicle  control  transitions  between  ground  and  on-board 
depending  on  mission  phase  and  particular  event  conditions: 

•  Before  Launch 

o  Ground  is  master 

o  Control  transitions  to  vehicle  during  launch  sequence 

•  During  Flight 

o  Vehicle  is  master  (autonomous) 
o  Ground  monitors  via  downlink  telemetry 
o  Ground  takes  control  when  appropriate 

•  Post  Landing 

o  Ground  is  master  (after  auto-safing) 


Command  Relationships  -  Maintenance  and  Logistics 

Maintenance  and  Logistics  can  be  considered  part  of  ground  control  (under  the 
overarching  domain  of  “Operations  Control  Center”)  or  a  separate  system  entirely.  Their 
relationship  to  the  vehicle  is  either  reactionary  or  scheduled  and  does  not  consist  of  a 
hierarchical  relationship. 


Interactions: 

•  Scheduled  Maintenance:  Based  on  flight  hours  and  is  performed  at  either  the 
base-level  or  at  a  depot.  Collected  historical  data  from  ISHM  monitoring  can  be 
used  to  highlight  components  that  need  to  be  inspected. 
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•  Unscheduled  Maintenance:  Initiated  when  a  fault  has  been  discovered.  Once  the 


ISHM  has  detected  an  anomaly,  the  appropriate  data  is  sent  to  Maintenance  and 
Logistic  for  action. 

•  Post  Mission:  Degradation  and  non-critical  fault  information  are  sent  to 
Maintenance  and  Logistics  to  improve  vehicle  turn-around  time. 

Command  Relationships  -  On-Board  Flight  Control 

The  on-board  flight  control  receives  command  to  execute  an  action  from  ISHM  generated 
by  either  ISHM  and/or  ground  C2.  The  autonomous  on-board  flight  control  will 
decompose  these  decisions  and  action  lists  into  a  set  of  commands  and  send  them  to  the 
appropriate  systems  for  execution.  On-board  flight  control  schedules  these  tasks 
accordingly  in  order  to  complete  in  the  prescribed  time. 

As  a  vehicle  system,  on-board  flight  control  health  status,  events,  time,  and  mission 
information  are  continuously  sent  to  ISHM.  ISHM  in  turn  continuously  provides  the 
vehicle  system  health  assessments,  vehicle  capability,  and  mitigation  actions 
predetermined  for  particular  anomalies. 

4.2.2  OV-5b  Operational  Activity  Models 

The  OV-5b  “Operational  Activity  Model”  shows  the  activity  flow  needed  for  the 

operation  of  a  typical  ISHM  system.  For  graphical  simplicity,  ISHM  has  been  divided 

into  three  main  activity  models:  the  first  (Figure  7)  being  the  activities  performed  under 

nominal  mission  operations;  the  second  (Figure  8)  concerning  the  actions  performed 
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when  a  fault  is  detected  during  a  mission;  and  the  third  (Figure  9)  the  activities  involved 
over  the  lifetime  of  the  UAS. 
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«0V-5»  act  OV-5  [0V-5b  Nominal  Mission  Operations] 


Mission 

Complete 


Figure  7  -  OV-5b  "Nominal  Mission  Operations' 
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Since  current  health  management  or  monitoring  technologies  also  use  sensors,  the 
activity  flow  through  the  diagram  in  Figure  7  is  generally  the  same  between  a  UAS  with 
ISHM  and  one  without.  The  difference  occurs  when  a  fault  is  detected,  without  ISHM 
there  is  no  certainty  as  to  what  is  actually  occurring  on  the  UAS  and  aside  from  a  few 
prevalent  and  simplistic  fault  conditions  the  UAS  will  be  recalled  to  base,  ending  the 
mission.  With  ISHM,  greater  system  awareness  is  achieved  and  alternative  mitigation 
actions  can  be  found  other  than  immediately  recalling  the  vehicle. 

The  OV-5b  diagram  “Fault  During  a  Mission”,  as  seen  in  Figure  8,  is  where  the  bulk  of 
ISHM  activities  are  performed.  Once  a  fault  (or  multiple  faults)  is  confirmed,  ISHM 
loops  through  the  diagnostic  and  prognostic  algorithms,  determining  the  type  and  location 
of  the  fault  as  well  as  calculating  the  estimated  time  to  failure.  This  data  is  then  pushed  to 
the  decision  reasoning  system,  where  the  faults  are  prioritized  and  the  remaining 
capability  of  the  vehicle  compared  to  the  current  mission  tasking.  Mitigation  actions  are 
then  selected  from  the  data  store  and  recommended  for  the  vehicle’s  autonomous 
command  and  control  system  to  evaluate.  Ideally,  the  command  and  control  system 
would  accept  the  mitigation  actions,  execute  them,  and  the  UAS  would  be  re-tasked  or 
would  continue  on  the  mission  as  appropriate.  A  situation  where  the  command  and 
control  system  would  not  accept  the  mitigation  actions  would  be  if  ISHM  recommended 
actions  that  would  cause  the  UAS  to  depart  flight;  although  this  is  unlikely,  the  hierarchy 
must  be  maintained  as  the  autonomous  command  and  control  system  is  flight-critical  and 
ISHM  is  not. 
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«OV-5»  act  OV-5  [0V-5b  Fault  During  Mission] 


Figure  8  -  OV-5b  "Fault  During  Mission” 
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The  activity  flow  through  the  OV-5b  diagram  “Lifetime  Operations”,  as  seen  in  Figure  9, 
also  parallels  UAS  without  ISHM  activities.  The  difference  would  be  found  in  the 
quantity  of  activities  performed,  ideally  a  system  with  ISHM  would  have  fewer 
scheduled  and  unscheduled  maintenance  actions. 

Theoretically,  a  system  with  ISHM  would  approach  condition-based  maintenance,  where 
all  maintenance  actions  are  driven  by  the  prognostic  and  diagnostic  modules,  eliminating 
scheduled  maintenance.  However,  current  ISHM  technologies  have  not  yet  reached  a 
level  of  confidence  where  scheduled  maintenance  can  be  entirely  removed  from 
maintenance  operations.  To  represent  how  ISHM  would  be  introduced  to  Air  Force 
operations  in  the  current  generation  of  technology,  scheduled  maintenance  remains  in  the 
architecture  as  a  health  management  action. 
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«0V-5»  act  OV-5  [0V-5b  Lifetime  Operations] 


Vehicle 

Retired 


Figure  9  -  OV-5b  "Lifetime  Operations" 
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4.2.3  OV-6a  Rules  Model 


Development  of  the  OV-6a  Rules  Model  closely  followed  the  development  of  the  OV-5b 
diagrams.  The  OV-6a  model,  seen  in  Figure  10,  represents  the  decisions  made  by  ISHM 
over  a  single  mission.  The  ISHM  metrics  that  drive  the  model  are  the  Probability  of 
Detection  (Pd),  the  Probability  of  a  False  Alarm  (Pfa),  and  the  Diagnostic  Confidence 
Level  (Dcl).  The  Probability  of  Detection  and  Probability  of  a  False  Alarm  are  dependent 
on  the  sensor  quality.  Generally,  a  higher  Probability  of  Detection  also  equates  to  a 
higher  Probability  of  False  Alarm.  The  Diagnostic  Confidence  Level  represents  the 
quality  of  the  diagnostic  and  prognostics  algorithms.  Better  algorithms  would  give  a 
higher  Diagnostic  Confidence  Level  and  therefore  a  better  probability  of  assigning  the 
correct  fault  category  for  a  detected  fault.  The  Rules  Model  logic  is  discussed  further  in 
Section  4.3. 
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Figure  10  -  OV-6a  "Rules  Model" 


4.2.4  SV-1  Systems  Interface  Model 

The  SV-1  Systems  Interface  Model  depicted  in  Figure  11  is  for  a  UAS  with  ISHM  using 
current  ISHM  technology.  In  this  architecture  ISHM  starts  at  the  subsystem  and 
component  level,  with  each  critical  subsystem  having  its  own  prognostic/diagnostic 
module.  Having  the  prognostic  and  diagnostic  module  at  this  lower  level  allows  each 
module  to  be  individually  configured  to  best  interpret  the  health  of  that  particular 
subsystem.  The  prognostic  and  diagnostic  module  ingests  data  from  the  sensors  (or 
sensor  suites,  depending  on  the  complexity  of  the  subsystem),  and  can  command  system 
effectors  when  investigating  off-nominal  conditions.  An  example  of  when  effectors  for  a 
subsystem  would  be  utilized  is  when  detecting  structural  cracks;  the  module  would  excite 
a  piezoelectric  transducer  (i.e.  an  effector),  which  would  send  out  an  elastic  wave  from 
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the  transducer,  the  wave  would  then  be  measured  by  sensors  further  down  the  component 
and  the  module  would  evaluate  the  data  for  any  deviations. 
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«SV-1»  composite  structure  SV-1  [SV-1] 


Figure  11  -  SV-1  "Systems  Interface  Model" 
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The  information  from  the  subsystem  and  component  level  is  then  fed  up  to  the  system 
level  to  an  Information  Management  System.  This  system  includes  the  decision  reasoner, 
a  data  store  with  a  list  of  safe  states  associated  with  known  fault  events  and  mitigation 
steps,  and  another  data  collection  module  to  store  health-related  data  and  to  process 
information  to  be  sent  out  as  heartbeats  (i.e.  periodic  health  state  assessments  to  ground- 
based  operations)  or  maintenance  actions,  as  appropriate.  In  the  case  of  a  fault,  the 
Information  Management  System  assesses  the  new  health  of  the  vehicles  (based  on  the 
estimated  time  to  failure  and  current  UAS  capabilities)  and  selects  the  mitigation  steps  to 
be  recommended  the  on-board  command  and  control  unit.  This  on-board  command  and 
control  unit  is  represented  by  the  Vehicle  Management  System  in  the  diagram. 

The  last  level  in  the  diagram  includes  the  systems  found  at  the  ground  level,  to  include 
the  Operations  Control  Center;  Ground  Operations  such  as  maintenance  and  logistics; 
and  a  ground  component  of  ISHM.  As  with  leaving  scheduled  maintenance  as  an  activity 
in  the  OV-5b,  the  current  state  of  ISHM  technology  does  not  allow  for  full  autonomy  in 
its  decision  making.  Given  time  to  review  (some  failures  will  be  too  imminent  to  allow 
time  for  review),  a  ground-based  operator  will  be  reviewing  the  activities  controlled  by 
ISHM,  separate  from  the  ground  control  center,  and  has  the  authority  to  override  ISHM 
commands  when  appropriate. 

4.3  ISHM  Analytic  Model 

Using  the  Rules  Model  created  in  Figure  10,  a  model  was  developed  to  simulate  the 
lifetime  of  a  UAS  and  the  effects  of  ISHM  on  the  mission  success  rate  and  expected 
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number  of  unscheduled  maintenance  actions.  The  model  parameters  were  displayed 
previously  in  Table  6. 

The  model  begins  by  generating  a  random  fault  time  (in  hours)  from  the  failure 
distribution  provided  by  the  user,  tFauit,  and  four  random  numbers  between  0  and  1 : 
RANDDetect.  RANDfa,  RANDcategory,  and  RANDcm-  The  model  then  determines  if  a  fault 
is  detected,  whether  or  not  a  fault  has  occurred,  or  if  a  fault  was  not  detected,  whether  or 
not  a  fault  has  occurred,  for  an  average  mission  length  (tM),  Probability  of  Detection  (PD), 
and  Probability  of  False  Alarm  (Pfa)  using  Equation  9. 

If  {t Fault  —  tM)  Then  Fault  =  1,  Else  Fault  —  0 
If  (Fault  =  ln  RANDDetect  <  PD )  Then  Detect  =  1 
If  (Fault  =  ln  RANDDetect  >  PD )  Then  Detect  =  0  (9) 

If  (Fault  =  0  n  RANDfa  <  PFA )  Then  Detect  =  1 
If  (Fault  =  0  n  RANDfa  >  PFA )  Then  Detect  —  0 

A  fault  category  is  then  assigned  using  RANDcategory  and  the  P(Fault  Categories) 
distribution  provided  by  the  user.  Equation  10  displays  how  this  category  is  assigned: 
If(RANDCategory  <  P(Fault  Cat  /))  Then  CategoryTRUE  =  1  *  Fault 
Elsel f  (RAN DCategory  <  P(Fault  Cat  //))  Then  CategoryTRVE  =  2  *  Fault 
ElseIf(RANDCategory  <  P(Fault  Cat  III))  Then  CategoryTrue  —  3  *  Fault 
Elsel f  (RAND  Category  <  P  (Fault  Cat  IV))  Then  Cate  gory  True  —  A*  Fault 

Else  CategoryTrue  —  5  *  Fault 
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A  confusion  matrix,  displayed  in  Table  8,  is  used  to  determine  the  declared  fault  category 
based  upon  CategoryTme-  The  confusion  matrix  initiates  using  the  diagnostic  confidence 
level,  Dcl,  as  the  basis,  but  the  model  al  lows  the  user  to  input  values  manually  if 
necessary. 


Table  8  -  Confusion  Matrix 


Confusion 

Matrix 

True  Fault  Category 

Declared 

Fault 

Category 

Nominal 

I 

II 

III 

IV 

V 

Nominal 

Dcl 

(1-Dcl)/2 

0 

0 

0 

0 

I 

1-Dcl 

Dcl 

(1-Dcl)/2 

0 

0 

0 

II 

0 

(1-Dcl)/2 

Dcl 

(1-Dcl)/2 

0 

0 

III 

0 

0 

(1-Dcl)/2 

Dcl 

(1-Dcl)/2 

0 

IV 

0 

0 

0 

(1-Dcl)/2 

Dcl 

1-Dcl 

y 

0 

0 

0 

0 

(1-Dcl)/2 

Dcl 

An  example  of  using  the  confusion  matrix  given  CategoryTme  =  II  can  be  seen  in 
Equation  11: 

If  (rANDcm  <  1  Then  CategoryDetect  —  I  *  Detect 

ElseIf(RANDCM  <  DCL)Then  CategoryDetect  —  II  *  Detect  (11) 

Else  CategoryDetect  —  III  *  Detect 
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Category-True  and  CategoryDetect  are  then  used  to  calculate  mission  success  rates  and 
maintenance  actions  using  the  formulas  found  in  Table  9.  For  this  research  effort, 
partially  completed  missions  are  considered  successful  missions. 


Table  9  -  Mission  Success  and  Maintenance  Rates  Formulas 


System  without  ISHM 

Formula 

Mission  Success? 

K 

jl,  if  CategoryDetect  =  0  AND  CategoryTme  <  2 
l _ 0,  otherwise 

Maintenance  Required? 

1,  if  CategoryDetect  =  1,  2,  3,  4 
otherwise 

System  with  ISHM 

Formula 

Mission  Success? 

-< 

1,  otherwise 

0,  if  CategoryDetect  >  4 

OR  CategoryTrue  >  3  when  CategoryDetect  <  2 
OR  CategoryTme  >  4  when  CategoryDetect  =  3 
OR  CategoryTme  =  5 

Maintenance  Required? 

I 

1,  if  CategoryDetect  =  2,  3,  4 

0,  otherwise 

The  model  then  outputs  the  number  of  missions  attempted,  number  of  missions 
completed  successfully,  and  number  of  unscheduled  maintenance  actions  initiated  for 
both  a  UAS  with  ISHM  and  without. 


This  model  has  several  assumptions  and  limitations  that  need  to  be  weighed  to  fully 
understand  how  the  results  can  be  used  by  decision  makers. 


•  Each  simulation  is  independent;  a  simulation  being  a  lifetime  of  the  UAS 

•  Sensor  and  system  degradation  effects  are  not  taken  into  account  in  this  model 
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•  The  addition  of  ISHM  causes  negligible  performance  degradation  of  the  UAS 

•  The  Probability  of  Detection  (Pd)  and  the  Probability  of  False  Alarm  (Pfa)  are  the 
same  for  a  UAS  without  ISHM  (using  current  health  management  practices)  and 
with  ISHM.  In  reality,  ISHM  would  have  additional  sensors  and  effectors  based 
on  the  results  of  the  FMECA,  resulting  in  a  different  Pd  and  Pfa. 

•  Any  fault  detected  will  result  in  a  cancelled  mission  under  current 
detection/health  management  capabilities 

•  The  scheduled  maintenance  intervals  act  as  a  renewal  process  -  that  is,  if  the  UAS 
reaches  a  scheduled  maintenance  interval,  the  vehicle  is  returned  to  a  “like  new” 
state. 

•  PD  and  PFa  are  representative  of  the  entire  suite  of  sensors  on  the  UAS.  In  reality, 
each  sensor  would  have  its  own  individual  performance  characteristics. 


The  model  was  coded  in  Microsoft  Excel©  using  Visual  Basic  Applications  (VBA)  for 
Excel;  the  full  code  can  be  found  in  Appendix  C:  Analytic  Model  Code. 

4.4  Model  Analysis 

As  stated  in  Chapter  III,  the  model  will  be  analyzed  using  Design  of  Experiments  (DOE) 
techniques  to  determine  the  region  where  ISHM  is  most  effective.  As  the  model  assumes 
that  the  sensor  characteristics  are  the  same  for  the  baseline  UAS  and  the  UAS  with 
ISHM,  the  model  is  best  used  to  evaluate  the  situation  where  ISHM  prognostic/diagnostic 
modules  and  the  information  management  system  would  be  attached  to  the  existing 
sensors  on  the  baseline  UAS.  The  DOE  techniques  used  in  the  section  are  taken  from 
Design  and  Analysis  of  Experiments  by  Douglas  C.  Montgomery  and  were  described  in 
detail  in  Section  2.7  Design  of  Experiments  [2009]. 
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4.4.1  Design  of  Experiments  Test  Design 

The  first  two  guidelines,  defining  the  problem  and  selecting  the  response  variable,  have 
been  discussed  in  depth  previously  in  this  section.  The  next  step  is  to  identify  the  design 
factors  and  their  appropriate  levels.  For  this  analytic  model,  there  are  14  separate  inputs 
that  can  be  used  as  design  factors,  as  shown  in  Table  10. 


Table  10  -  Model  Input 


UAS  Properties 

Model  Input 

P(Failure) 

Lailure  Distribution  (i.e.  Normal);  two 
parameters  (i.e.  p  and  a) 

P(Fault  Categories) 

P(Lault  Category  1) 

P(Lault  Category  2) 

P(Lault  Category  3) 

P(Lault  Category  4) 

P(Lault  Category  5) 

*sum  of  these  probabilities  must  add  to  1 

Average  Mission  Length 

tm 

Scheduled  Maintenance 

Interval 

tpm 

Expected  System  Lifetime 

T 

ISHM  Properties 

Model  Input 

Probability  of  detecting  a  fault 

Pd 

Probability  of  the  sensor 
reading  a  false  alarm 

Pfa 

Diagnostic  Confidence  Level 

Dcl 

As  discussed  in  Chapter  III,  several  of  these  factors  will  be  fixed  and  will  be  used  to 
approximate  a  typical  UAS:  failure  distribution,  tM,  tpM,  and  T.  The  remaining  factors 
then  become  the  design  factors  and  will  be  varied  to  investigate  their  effect  on  ISHM 
effectiveness. 
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A  key  factor  in  DOE  is  independence  in  the  factors  being  investigated.  This  is  not 
possible  for  two  groups  of  the  factors:  P(Fault  Categories),  which  must  sum  to  one;  and 
the  sensor  characteristics  Pd  and  Pfa,  which  are  dependent  on  each  other.  Instead  of  using 
all  five  P(Fault  Categories),  two  will  be  selected  to  represent  this  group.  P(Fault  Category 
II)  and  P(Fault  Category  III)  best  reflect  the  difference  in  how  mission  success  is 
calculated  in  the  model  for  a  UAS  with  ISHM  and  for  one  using  current  health 
management  practices  (see  Table  9). 


The  sensor  performance  characteristics,  Pd  and  Pfa  are  determined  by  Receiver 
Operating  Characteristic  (ROC)  curves,  which  relate  true  positive  fraction  to  false 
positive  fraction.  The  ROC  curve  model  used  in  this  research  is  shown  in  Equation  12 
and  is  derived  from  [Moses,  1993]: 


(1  -  Pfa)  = 


(1  -  c)PD  +  c 

where  the  parameter  c  e  [  1  ,oo]  represents  the  quality  of  the  sensor;  as  c  increases,  the 
ROC  improves,  as  c  — *  oo,  the  area  under  the  curve  approaches  unity  indicating  perfect 
classification.  There  are  many  ways  to  calculate  c  but  for  the  purposes  of  this  model  no 
specific  equation  will  be  provided,  c  will  instead  represent  a  general  quality.  A  family  of 
ROC  curves  is  presented  in  Figure  12. 


(12) 
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Figure  12  -  Family  of  ROC  Curves 


To  break  the  dependence  on  each  other,  only  PD  and  sensor  quality  (c)  will  be  evaluated 
for  the  analysis. 


The  initial  high  (+1)  and  low  (-1)  discrete  settings  for  each  of  the  seven  factors  were 
chosen  with  input  from  health  management  Subject  Matter  Experts  at  AFRL/RQ  and  are 
displayed  in  Table  11.  Center  points  are  also  included,  as  they  are  necessary  to  check  for 
curvature  in  the  response  surface. 
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Table  11  -  DOE  Factor  Levels 


Factor 

Discrete  Settings 

-1 

Center 

+1 

Weibull  -  Theta 

700 

850 

1000 

Weibull  -  Beta 

2.5 

2.75 

3 

Sensor  Quality 

100 

300 

500 

Pd 

0.3 

0.6 

0.9 

P(Fault  Category  II) 

0.1 

0.25 

0.4 

P(Fault  Category  III) 

0.1 

0.25 

0.4 

Dcl 

0.6 

0.75 

0.9 

Factor 

Fixed  Settings 

Distribution 

Weibull 

T 

10,000  hrs 

tM 

10  hrs 

tpM 

1,000  hrs 

n 

To  test  every  combination  of  high/low  factors  in  a  factorial  design,  a  2  design  would 
require  at  least  128  runs,  not  including  the  additional  center  points  and  any  replications. 
With  this  in  mind,  a  fractional  factorial  27  4  Resolution  III  design  with  two  replicates  and 
four  center  points  was  chosen,  for  a  total  of  28  runs.  Each  run  would  also  include  four 
repeated  measurements  (i.e.,  four  Monte  Carlo  trials)  for  a  total  of  12  measurements  for 
each  test  point  selected.  The  high  number  of  measurements  for  each  test  point  was  chosen 
due  to  Excel’s  inadequacies  at  random  number  generation.  Previous  research  into  Excel 
has  shown  that  Excel’s  random  number  generation  does  not  fulfill  the  basic  requirements 
for  a  random  number  generator  to  be  used  for  scientific  purposes  [McCullough,  2008] . 
Since  the  model  relies  on  primarily  on  the  random  number  generator,  a  large  number  of 
measurements  for  each  test  point  will  hopefully  assuage  the  number  generation  problems. 


The  defining  relationship  for  this  experiment  was  chosen  to  alias  higher  order  effects  and 

focus  on  the  main  factors  and  low-order  interactions,  following  the  sparsity  of  effects 
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principle  as  discussed  in  Section  2.7: 1  =  ABD  =  ACE  =  BCF  =  ABCG.  For  this 
relationship  A  =  Weibull-Theta,  B  =  Weibull-Beta,  C  =  Sensor  Quality,  D  =  Probability 
of  Detection,  E  =  P(Fault  Category  II),  F  =  P(Fault  Category  III)  and  G  =  the  Diagnostic 
Confidence  Fevel.  The  full  alias  structure  can  be  found  in  Appendix  D:  Design  of 
Experiments  Results  and  Models. 

4.4.2  Design  of  Experiments  Results  and  Conclusions 

The  full  experiment  with  test  design,  results,  and  statistical  analysis  can  be  found  in 
Appendix  D:  Design  of  Experiments  Results  and  Models.  A  summary  of  the  results  and 
the  corresponding  conclusions  are  detailed  in  this  section.  The  statistical  analysis  in  this 
section  was  performed  using  JMP®  Version  9.0.1. 

One  of  the  main  results  is  that  not  all  of  the  design  factors  are  significant.  Using  an  F-test, 
only  four  main  factors  -  Weibull-O,  Pd,  P(Fault  Category  III),  and  sensor  quality  -  and 
some  low-order  interactions  were  found  to  significantly  affect  the  response.  The 
remaining  factors  can  essentially  be  ignored  when  using  the  model  to  compare  a  UAS 
with  ISHM  and  without.  Effect  tests  on  the  significant  factors  and  interactions  can  be 
found  in  Figure  13,  the  alpha  level  for  the  significance  tests  was  0.05.  The  model  was 
also  found  to  include  quadratic  terms,  in  this  case  Weibull-0  *  Weibull-0  and  P(Fault 
Category  III)*  P(Fault  Category  III),  which  indicated  a  second-order  response  surface 
model  and  that  some  curvature  would  be  seen  in  the  response  surface. 
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Source 

Prob  Detection 
Prob  Fault  Category  III 
Weibull  -  theta 
Sensor  Quality 

Prob  Fault  Category  lll*Prob  Fault  Category  III 

Prob  Detection*Weibull  -  theta 

Weibull  -  theta*Weibull  -  theta 

Prob  Fault  Category  lll*Sensor  Quality 


Nparm 

DF 

Sum  of 
Squares 

F  Ratio 

Prob  >  F 

1 

1 

976.8580 

48.8474 

<0001* 

1 

1 

383.4336 

19.1734 

<0001* 

1 

1 

225.5012 

11.2761 

0.0018* 

1 

1 

91.5769 

4.5793 

0.0390* 

1 

1 

238.7926 

11.9407 

0.0014* 

1 

1 

954.2876 

47.7187 

<0001* 

1 

1 

108.6350 

5.4322 

0.0253* 

1 

1 

1482.2742 

74.1205 

<0001* 

Figure  13  -  Effect  Tests  on  Significant  Factors  and  Interactions 


The  final  model  equation,  displayed  in  Equation  13,  mapping  the  response  surface  was 
determined  to  be: 

y  =  (-5.861  +  0.0150  +  17.322PD  -  67.369 P {Fault  Cat  III )  + 

0.007c  +  50 1.715  (P(FaitZt  Cat  III )  -  0.191)2  - 
0.237(0  -  850)(Pd  -  0.652)  -  0.0001(0  -  850)2  + 

0.391)  -  0.41 5 (P (Fault  Cat  III )  -  0.191)(c  -  300))2 

where  y  is  the  difference  between  the  number  of  successful  missions  calculated 

for  a  system  with  ISHM  and  the  same  system  without  ISHM  (i.e.  using 

current  health  management  techniques) 


From  this  equation  the  stationary  point  is  a  region  of  minimum  response,  clearly  visible 
in  Figure  14. 


(13) 
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1000 


—4000 


We*ull  -  theta 


Figure  14  -  Response  Surface  for  Analytic  Model 


While  this  response  surface  best  illustrates  the  region  were  the  response  is  at  its 
minimum,  or  the  region  where  adding  ISHM  to  the  UAS  baseline  would  not  significantly 
affect  mission  success  rates,  there  can  be  some  inferences  made  about  the  regions  that 
maximize  the  response.  By  not  determining  the  ISHM  performance  characteristic  (the 
Diagnostic  Confidence  Level)  significant,  this  evaluation  implies  that  the  benefits  or 
disadvantages  of  adding  ISHM  rely  primarily  on  the  performance  of  the  baseline  health 
management  system.  Specifically,  that  ISHM  becomes  more  beneficial  as  the  baseline 
health  management  system  performs  worse. 
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Another  useful  result  of  this  analysis  is  that  the  model  equation  can  be  used  to  test  if 
adding  ISHM  to  a  UAS  will  statistically  affect  the  mission  success  rates.  This  can  be 
done  using  a  two-sample  t-test,  because  we  can  assume  that  the  variance  is  equal  between 
mission  success  rates  calculated  for  the  UAS  with  ISHM  and  the  UAS  without.  The  t-test 


where  yl  ~  Yi  =  the  output  of  the  model  equation,  the  difference  between  the 
number  of  successful  missions  calculated  for  a  system  without 
ISHM  and  the  same  system  with  ISHM 
Sp  =  sample  variance. 
n  =  population  size. 


The  Mean  Square  Error  (MSe)  calculated  for  the  model  can  be  used  as  an  estimate  of 

sample  variance.  The  sample  size  used  to  create  the  model,  in  this  case  46  trials  with  four 

repeated  measurements  for  each  trial,  can  be  used  as  the  population  size.  The  addition  of 

ISHM  would  be  considered  statistically  significant  if  |t0|  >  ta  where  a  is  the  level 

2’^ 

of  significance.  Using  the  model  results  detailed  in  Appendix  D  and  an  alpha  of  0.05,  the 
updated  Equation  15  becomes: 

yl-y^  _Y\~Y2 

0  HT  2.4026 

23'045Jl84  (15) 

£“;2n-2  =  *-0.025,366  =  1-967 
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Using  Equation  12  it  can  be  inferred  that  if  the  difference  between  the  expected  number 


of  successful  missions  calculated  for  a  system  without  ISHM  and  the  same  system  with 
ISHM  is  greater  than  4.726,  then  the  addition  of  ISHM  to  the  baseline  UAS  will  result  in 
a  statistically  significant  difference  in  mission  success  rates.  Since  the  mission  success 
rate  difference  is  always  positive,  the  addition  of  ISHM  can  be  considered  a  beneficial 
addition  in  terms  of  mission  success  rates. 

4.5  Summary 

This  chapter  covered  the  final  products  and  results  from  the  methodology  presented  in 
Chapter  III.  An  analytic  architecture  was  created  using  the  Architecture  Based  Evaluation 
Process  and  then  evaluated  using  Design  of  Experiments  techniques.  Results  from  the 
evaluation  indicated  that  installing  ISHM  in  existing  UAS  platforms  is  only  worthwhile, 
in  terms  of  mission  effectiveness,  when  the  existing  UAS’s  health  management  system 
has  significant  detection  and  false  alarm  problems.  The  products  and  model  results  will 
form  the  basis  of  this  research  effort’s  conclusions  and  recommendations  discussed  in 
Chapter  V. 
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V.  Conclusions  and  Recommendations 


5.1  Chapter  Overview 

This  chapter  will  answer  the  research  objectives  and  discuss  areas  for  future  research. 

5.2  Research  Questions  Answered 

The  focus  of  this  research  effort  was  to  quantify  the  mission-related  benefits  of  ISHM  by 
constructing  architecture  for  analysis  to  compare  against  current  autonomous  vehicle 
capabilities,  and  to  provide  a  general  baseline  model  that  can  be  implemented  over  any 
current  or  future  autonomous  vehicle.  To  do  this,  a  literature  review  was  conducted  to 
answer  the  following  questions,  posed  initially  in  Chapter  I: 

What  is  the  current  status  of  UAS  health  management? 

The  Air  Force  UAS  programs  currently  use  independent  sensors  incorporated  into  the 
vehicle’s  hardware  to  monitor  for  fault  indicators  on  critical  subsystems.  The  sensor  data 
is  continuously  transmitted  to  ground  operations  where  it  is  then  processed  to  detect 
anomalies.  If  the  data  indicates  a  fault  has  occurred  the  ground  operator  will  execute  pre¬ 
determined  mitigation  steps,  dependent  on  which  sensor  indicated  a  fault,  and  relay  a 
message  to  the  maintenance  and  logistics  element.  Once  the  vehicle  lands,  maintenance 
personnel  perform  diagnostic  tests  to  confirm  the  location  and  identify  the  type  of  fault, 
and  then  perform  maintenance  actions  to  restore  the  component.  This  is  less  health 
management  than  health  monitoring,  in  terms  of  nomenclature. 
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What  are  the  essential  elements  ofISHM? 

A  typical  ISHM  system  consists  of  sensors  placed  at  critical  components  within 
subsystems  of  the  vehicle  that  stream  data  to  a  management  system.  The  management 
system  processes  the  sensor  data,  executes  diagnostic  and  prognostic  algorithms,  and  then 
feeds  this  information  through  a  reasoner,  as  previously  displayed  in  Figure  3.  This 
management  system  can  either  be  on-board  the  vehicle  in  a  hardware  configuration  or 
off-board  enabling  the  ground  command  and  control  (C2)  element. 


Management  System 


Figure  3  -  Typical  ISHM  configuration  [Benedettini,  2009] 


Sensors  can  be  conventional  or  specifically  tailored  to  health  management  applications 
The  sensor  data  is  then  processed  to  remove  any  artifacts  or  noises  and  manipulated  to 
extract  fault  features.  The  diagnostic  module  then  analyzes  the  fault  features  to  detect, 
identify,  and  isolate  developing  failure  conditions.  The  diagnostic  information  will  be 
combined  with  historical  data  in  the  prognostic  module  to  generate  an  estimation  of 
failure  times.  Finally,  the  diagnostic  and  prognostic  information  is  turned  over  to  the 
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reasoner  module  which  analyzes  available  resources,  decides  which  hazard  mitigation 
steps  to  execute,  and  then  passes  the  selected  decision  to  the  on-board  C2  module  and 
relays  appropriate  information  to  the  ground  C2  operator  and  maintenance  element 
[Benedettini,  2009] . 

What  are  the  expected  benefits  ofISHM? 

There  were  five  areas  that  were  determined  to  benefit  the  most  from  adding  ISHM  to 
UAS  platforms:  rate  of  scheduled  and  unscheduled  maintenance,  repair  times,  operational 
availability,  mission  success,  and  cost.  With  ISHM  implemented,  the  probability  of 
unscheduled  maintenance  should  decrease  as  unscheduled  maintenance  becomes  more 
fault  driven  and  scheduled  maintenance  intervals  can  also  be  investigated  for  potential 
relaxation  or  removal.  Ideally,  the  aircraft  would  replace  time-based  or  event-driven 
maintenance  with  a  condition-based  maintenance  system,  where  maintenance  is  only 
performed  based  on  objective  evidence  of  actual  or  predictable  failure  of  a  system  or  its 
components  [OSAIDD,  1999].  Repair  times  would  decrease  as  adding  prognostic 
technology  would  result  in  each  subsystem  or  component  having  an  estimated  time  to 
failure.  Knowing  which  systems  are  near  failure  ahead  of  time  would  again  allow  the 
maintenance  element  time  to  pre-position  the  necessary  maintenance  equipment  and 
personnel  or  order  any  necessary  replacement  parts. 

Based  on  the  decreased  downtime  in  unscheduled  maintenance  and  scheduled 
maintenance  from  ISHM,  the  operational  availability  for  each  aircraft  should  improve. 
Another  factor  affecting  operational  availability  is  mission  turn-around  time,  or  the  time 
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from  when  the  aircraft  lands  to  when  it  is  ready  for  the  next  mission.  Without  ISHM, 
mission  turn-around  time  can  include  lengthy  inspection  tests  to  check  for  failures.  Since 
these  tests  are  not  needed  with  continuous  monitoring  and  a  higher  confidence  in  fault 
detection,  the  mission  turn-around  time  should  decrease,  increasing  operational 
availability.  Mission  success  rates  would  also  theoretically  increase  as  having  situational 
awareness  of  the  entire  health  state  of  the  vehicle  assists  ground  operations  in  providing 
full  mission  coverage.  If  a  UAS  autonomously  detects  a  fault  and  due  to  the  fault 
criticality  (for  example,  low  fuel  levels)  decides  to  re-task  to  a  closer  trajectory,  ground 
operations  can  re-task  other  UAS  vehicles  to  ensure  coverage  of  the  priority  targets. 
Without  ISHM,  a  fault  alert  would  generally  give  no  indication  of  the  remaining 
performance  capability  of  the  UAS,  leaving  ground  operations  to  conservatively  scrap 
that  particular  mission  set.  ISHM  can  also  theoretically  expand  the  flight  envelope  of  the 
aircraft,  which  could  allow  for  larger  mission  sets. 

The  previous  benefits  all  have  some  measure  of  cost  savings  attached  to  them.  Having  a 
lower  total  maintenance  downtime,  due  to  decreases  in  scheduled  maintenance  and  a 
lower  probability  of  unscheduled  maintenance,  leads  to  a  lower  personnel  cost  and  even 
an  option  of  having  less  maintenance  personnel  needed.  Fewer  maintenance  actions  also 
indicate  a  potential  reduction  in  spares  and  supply  costs.  However,  there  is  an  inherent 
cost  in  implementing  ISHM,  not  just  to  the  vehicle  but  to  the  resulting  operational 
infrastructure.  The  cost  savings  must  be  weighed  against  the  implementation  costs  to 
truly  investigate  the  financial  aspect  of  ISHM. 
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The  answers  to  the  literature  review  questions  were  then  used  to  develop  an  analytic 
architecture  that  would  answer  these  primary  research  questions: 

What  are  the  potential  impacts  to  ground  control  stations  and  users? 

With  the  addition  of  ISHM,  unmanned  aerial  systems  (UAS)  move  closer  to  a  state  of 
true  autonomy  and  less  reliance  is  placed  on  ground  control  stations.  As  with  any  new 
technology,  a  phased  approach  would  be  appropriate  when  integrating  this  technology 
with  current  practices. 

The  architecture  built  for  this  effort  is  designed  for  the  initial  phase  and  resembles  a  state 

of  flexible  autonomy,  where  command  and  control  (C2)  of  the  UAS  shifts  from 

autonomous  to  operator  based  on  mission  phase  and  particular  event  conditions.  In 

general,  ground  C2  (as  represented  by  the  Operations  Control  Center  in  the  SV-1) 

commands  the  vehicle  before  launch  and  post  landing,  and  the  autonomous  C2  takes  over 

during  the  launch  sequence  and  releases  command  during  auto-safing.  Currently,  the 

ground  C2  still  maintains  significant  control  through  the  whole  flight,  even  though  the 

autonomous  capability  is  there.  ISHM  should  help  to  increase  the  level  of  autonomy 

within  future  UAS  since  ISHM  would  provide  an  estimation  of  the  system's  current 

abilities  to  enable  real-time  decision  making  by  the  vehicles  C2.  If  designed  for  some 

UAS  platforms,  such  as  the  RQ-4  Global  Hawk,  ground  C2  would  consist  of  separate 

Launch  and  Recovery  (LRE)  and  Mission  Control  Elements  (MCE).  Also  depending  on 

the  UAS,  ground  C2  can  have  the  ability  to  control  multiple  vehicles  at  a  single  time.  So 

far,  this  is  not  structurally  any  different  from  current  UAS  operations  as  performed  by  the 
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United  States  Air  Force.  Implementing  ISHM  into  the  UAS  concept  of  operations  would 
not  eliminate  any  of  the  current  ground  C2  infrastructure  but  would  instead  require  the 
addition  of  another  element,  the  ISHM  Ground  Station,  whose  sole  purpose  is  to  monitor 
and  verify  the  decisions  made  by  ISHM.  This  element  would  not  have  personnel  attached 
to  it,  it  is  instead  another  computer  or  set  of  computers  with  the  more  complex  algorithms 
that  would  not  be  able  to  stored  on  the  aircraft  due  to  the  processing  speed  limitations. 
ISHM  would  also  affect  current  users  on  the  ground  by  potentially  increasing  the  number 
of  vehicles  that  can  be  controlled  at  once;  with  health  management  handed  over  to  the 
vehicle,  ground  C2  has  the  ability  to  potentially  manage  more  UAS.  Additional  human 
factors  analysis  would  be  completed  to  determine  the  maximum  amount  of  vehicles  that 
ground  C2  can  safely  control. 

Ideally  in  the  next  phase  (as  confidence  in  ISHM  and  autonomous  technology  increases), 
the  entire  mission  from  launch  to  recovery  would  become  fully  automated,  with  ground 
C2  only  managing  the  mission  taskings  or  re-taskings.  Ground  operations,  previously 
managed  by  multiple  elements,  such  as  the  LRE,  MCE,  and  ISHM  Ground  Station,  can 
potentially  be  combined  into  one  center.  This  could  significantly  lower  the  amount  of 
personnel  needed  to  operate  a  UAS. 

What  are  the  potential  impacts  on  current  maintenance  practices? 

The  end-goal  of  ISHM  is  a  state  of  condition-based  maintenance,  where  maintenance  is 
only  performed  based  on  objective  evidence  of  actual  or  predictable  failure  of  a  system  or 
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its  components  [OSAIDD,  1999].  Mirroring  the  impact  on  ground  control  stations,  the 
changes  to  maintenance  practices  should  take  a  phased  approach. 

The  initial  phase  of  ISHM  implementation,  as  built  in  the  architecture,  closely  resembles 
current  practices.  There  are  still  scheduled  maintenance  intervals;  however,  by  providing 
continuous  monitoring  and  knowing  the  Estimated  Time  to  Failure  for  the  critical 
components,  these  intervals  have  the  potential  to  be  relaxed.  The  other  main  impact 
would  be  in  the  response  to  faults.  Before,  time-intensive  Built-in  Test  (BIT)  units  would 
be  used  to  verify  that  the  fault  exists  and  to  pinpoint  which  component  to  repair.  ISHM 
verifies  the  fault  in  flight  and  provides  reams  of  data  to  the  maintenance  element  for  their 
own  verification,  negating  the  use  of  the  BIT  unit.  Also,  by  knowing  the  specific  systems 
to  be  inspected  or  repaired  ahead  of  time,  the  maintenance  element  has  time  to  pre¬ 
position  the  necessary  equipment  and  personnel  or  order  any  necessary  replacement  parts 
before  the  UAS  has  completed  its  mission. 

The  next  phase  would  involve  upgrading  to  condition-based  maintenance.  Scheduled 
maintenance  intervals  would  no  longer  exist  and  the  entire  concept  of  operations  for 
maintenance  would  become  reactionary. 

What  are  the  performance  characteristics  necessary  for  ISHM  to  effect  mission 
success? 

A  response  surface  was  modeled  for  a  UAS  with  an  expected  lifetime  of  10,000  hours, 

maintenance  interval  of  1,000  hours,  and  average  mission  length  of  10  hours.  The  final 
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model  equation,  initially  shown  in  Equation  13,  mapping  this  surface  was  determined  to 


be: 


y  =  (-5.861  +  0.0150  +  17.322PD  -  67 . 369 P (Fault  Cat  III )  + 

0.007c  +  501.715(P(Fait/t  Cat  III )  -  0.191)2  -  (13) 

0.237(0  -  850) (PD  -  0.652)  -  0.0001(0  -  850)2  +  0.391)  - 
0.415(P(Pau/t  Cat  III )  -  0.191)(c  -  300))2 


where  y  is  the  difference  between  the  number  of  successful  missions  calculated 


for  a  system  with  ISHM  and  the  same  system  without  ISHM  (i.e.  using 


current  health  management  techniques) 


Contour  plots  for  the  response  surface  near  this  point  are  provided  in  Figure  15.  The 
statistical  analysis  performed  in  Section  4.4.2  determined  that  a  response  greater  than 
4.726  indicated  a  statistically  significant  difference  in  mission  success  rates.  The  shaded 
regions  on  the  contour  plots  indicate  areas  where  ISHM  is  not  beneficial.  If  the  factors 
fall  anywhere  outside  of  this  region,  ISHM  should  be  investigated  as  a  beneficial  addition 
to  the  existing  UAS  in  terms  of  mission  success  rates. 
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Prob  Detection  ..  Sensor  Quality 


Figure  15  -  Contour  Plots  for  Response  Surface 
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The  architecture  also  contributed  to  answering  these  secondary  questions: 

How  should  ISHM  data  be  presented  to  be  effective?  How  will  the  presentation  change 
in  regards  to  the  different  users  of  ISHM  (operators,  maintenance,  etc.)? 

As  seen  in  the  OV-2,  displayed  in  Figure  16,  there  are  several  types  of  information  that 
are  passed  from  ISHM  to  ground- level  operations:  vehicle  status,  vehicle  capabilities,  and 
maintenance  reports.  Additional  human  factors  research  will  be  needed  to  determine  how 
this  information  is  presented  to  the  users;  in  this  architecture  there  are  three  main  users  of 
the  data:  the  Operations  Control  Center  (OCC),  the  ISHM  Ground  Station  (in  the  OV-2, 
the  OCC  and  ISHM  Ground  Station  are  combined  under  “Ground  Command  and 
Control”),  and  Ground  Recovery  Operations  consisting  of  maintenance  and  other  launch 
and  recovery  operations.  While  maintenance  reports  are  unique  to  Ground  Recovery 
Operations,  the  OCC  and  the  ISHM  Ground  Station  exploit  the  vehicle  status  and 
capabilities  differently;  consideration  of  this  point  must  be  taken  when  researching  the 
best  way  to  present  the  data  to  the  personnel  of  each  element. 
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Figure  16  -  OV-2  “Operational  Node  Connectivity  Description” 


Is  ISHM  cost  effective? 

The  main  result  of  the  model  evaluation  indicated  that  the  quality  of  sensors  will  affect 
the  cost  and  mission  benefits  relative  to  the  degree  of  ISHM  implemented  on  a  system.  A 
cursory  interpretation  of  this  analysis  result  infers  that  decision  makers  should  compare 
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the  cost  and  mission  benefits  of  upgrading  the  sensors  with  the  cost  and  benefits  of 


implementing  ISHM;  however,  a  complete  cost  and  mission  benefit  analysis  should  be 
completed  before  making  any  conclusions  and  will  require  a  more  in-depth  ISHM  model 
than  that  presented  in  this  paper.  The  model  presented  in  this  research  effort  lays  the 
foundation  to  develop  the  more  in-depth  model. 

While  no  cost  data  was  included  in  the  model,  the  output  from  the  model  can  be  also  used 
when  evaluating  the  total  financial  benefit  of  ISHM.  By  putting  a  cost  on  an  average 
unscheduled  maintenance  action,  the  expected  number  of  maintenance  actions,  as  output 
by  the  model,  for  a  UAS  without  ISHM  and  one  with  ISHM  can  be  compared.  The  model 
can  also  be  used  to  determine  the  effect  of  longer  scheduled  maintenance  intervals  on 
expected  unscheduled  maintenance  actions  for  a  UAS  with  ISHM.  The  cost  saved  by 
having  longer  scheduled  maintenance  intervals  can  be  added  to  the  financial  evaluation 
for  decision  makers. 

The  expected  mission  success  rate  can  also  be  used  to  decide  whether  ISHM  is  cost 
effective.  There  is  a  cost  associated  with  preparing  a  UAS  for  launch  and  with  the 
recovery  actions  once  the  UAS  has  landed.  If  a  UAS  would  have  to  curtail  its  mission  or 
cancel  it  entirely  because  of  health  management  issues,  another  UAS  would  be  tasked  to 
complete  the  mission,  and  could  incur  additional  launch  and  recovery  costs  if  it  had  to  be 
launched  from  scratch.  The  expected  mission  success  rate  for  a  UAS  with  ISHM  and  for 
one  without  ISHM  could  be  quantified  as  an  expected  cost  per  mission  and  then 
compared  for  evaluation  by  decision  makers. 
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5.3  Recommendations  for  Future  Research 


There  are  numerous  opportunities  for  further  research  into  this  aspect  of  Integrated 
System  Health  Management.  A  large  benefit  to  the  research  would  be  lifting  some  of  the 
assumptions  under  which  the  analytic  model  operates.  One  large  assumption  is  that  the 
scheduled  maintenance  intervals  act  as  a  renewal  process;  this  is  not  close  to  reality,  as 
the  system  will  degrade  over  time,  even  with  adequate  maintenance  intervals.  A  second 
model  assumption  follows  along  the  same  lines,  only  in  this  case  assuming  that  there  are 
no  sensor  degradation  effects  over  the  lifetime  of  the  vehicle.  Over  time,  the  probability 
of  detection  for  the  sensor  will  decrease  and/or  the  probability  of  a  false  alarm  will 
increase.  Another  large  assumption  is  that  ISHM  uses  the  same  sensor  suite  that  is 
currently  in  the  baseline  UAS;  following  the  results  of  the  FMECA,  ISHM  would 
actually  supplement  the  original  health  management  or  monitoring  system  with  additional 
sensors  and  effectors.  The  model  should  be  updated  to  reflect  these  changes;  this  will 
give  a  more  accurate  representation  of  the  reliability  and  health  management  aspects  of 
the  baseline  UAS. 

Much  of  this  research  effort  used  theoretical  values  when  evaluating  the  model  and 
mapping  the  response  surface.  If  actual  failure  data  for  current  UAS  platforms  or 
information  becomes  available  for  commercially-implemented  ISHM  systems,  this 
information  can  be  fed  into  the  model  and  the  response  surface  can  be  re-evaluated. 

Large  ranges  were  used  for  the  theoretical  values  to  try  and  cover  a  variety  of  potential 
UAS  platforms  and  ISHM  systems;  this  leads  to  potential  model  inadequacy  because 
local  maximum  and  minimum  ridges  may  not  have  been  discovered.  Actual  ISHM 
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performance  data  will  hopefully  give  much  smaller  ranges  and  a  more  robust  response 
surface  can  be  determined. 

5.4  Summary 

In  this  research  effort,  an  analytic  architecture  was  created  to  help  determine  the  effect 
ISHM  had  on  mission  success  rates  for  a  UAS.  The  final  products  revealed  that,  for 
mission  success  rates  only,  ISHM  is  beneficial  in  situations  where  the  theoretical  UAS 
has  serious  problems  with  detection  and  false  alarm  rates.  Using  representative  data  for  a 
UAS,  the  analysis  determined  that  the  failure  distribution  parameters,  sensor  quality 
(which  determines  the  relationship  between  probability  of  detection  and  probability  of 
false  alarm),  and  probability  of  an  imminent  fault  during  a  mission  were  significant  to  the 
model.  The  result  of  the  model  determined  that  ISHM  can  result  in  a  significant 
improvement  on  mission  assurance,  especially  when  implemented  with  higher  quality 
sensors  and  on  vehicles  where  the  probability  of  imminent  failure  is  higher  relative  to  the 
mission  times  and  time  between  preventative  maintenance.  This  appears  consistent  with 
the  premise  that  ISHM  can  support  an  extension  of  preventative  maintenance  intervals 
with  an  attendant  reduction  in  sustainment  cost. 

It  is  important  to  note  that  the  analytic  model  had  several  broad  assumptions  that  affect 
these  conclusions:  (1)  the  model  assumed  that  ISHM  would  use  the  same  sensor  suite  that 
is  currently  in  the  baseline  UAS  -  this  does  not  reflect  reality,  ISHM  would  have 
additional  sensors  and  effectors  based  on  the  results  of  the  FMECA,  resulting  in  a 
different  PD  and  PFa;  (2)  the  model  is  limited  to  detecting  faults  that  the  current  system  is 
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looking  for  -  theoretically,  ISHM  would  gather  data  over  the  lifetime  of  the  vehicle  to 
supplement  these  fault  states  as  new  information  becomes  available;  (3)  the  model  does 
not  allow  for  system  or  sensor  degradation  -  this  negates  a  lot  of  the  benefits  provided  by 
prognostics.  Additional  analysis  is  needed  to  further  study  the  effect  of  ISHM  on  mission 
effectiveness.  These  results  should  also  be  taken  as  just  one  part  of  the  “big  picture”  of 
ISHM,  and  should  be  weighed  against  the  other  benefits  that  ISHM  provides. 
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Appendix  A:  Architecture-Based  Evaluation  Process  (ABEP) 


1.  Design  Operations  Concept  of  system  to  be  evaluated. 

Ops  concept  provides  the  system  description  which  the  architecture  will  model, 
and  the  models  will  simulate/evaluate. 

2.  Identify  Measures  of  Effectiveness  (MOE)  relevant  to  the  decision/evaluation 

Identify  the  metrics  that  represent  the  effectiveness  of  the  system. 

3.  Identify  required  level  of  abstraction  for  architecture  to  show  traceability  to 
MOE’s 

Analyze  the  Ops  Concept  to  determine  if  MOE’s  are  measured  at  the  output  of  the 
system,  within  the  system  (requiring  ‘drilling’  into  the  system  activities),  or  at  the 
output  of  activities  external  to  the  system  (requiring  external  systems  diagram) 

4.  Identify  architecture  views  necessary  to  capture  structure/relationships 

a.  Structure  (OV-1,  OV-2,  and  OV-5)  In  order  to  first  develop  the  structure  of  the 

analysis,  nearly  all  evaluations  will  require  the  OV-1  (High  Level 
Operations  Concept),  OV-2  (Operational  Node  Connectivity  Description), 
and  OV-5  Operational  Activity  Model  views.  The  level  of  abstraction  (A- 
1,  A-0,  AO  etc.)  of  the  OV-5  is  initially  identified  in  the  previous  step. 

b.  Decision  Logic  (OV-6a)  to  capture  the  logic  of  the  system,  nearly  all 

evaluations  will  require  the  OV-6a  Rules  Model,  developed  to  match  the 
level  of  abstraction  used  for  the  OV-5’s. 

c.  As  Required:  SV-2,  SV-4,  SV-7,OV-6b,  OV-6c  Depending  on  the  complexity, 

consideration  for  time  and  dependency  on  internal  performance  inputs, 
some  or  all  of  the  listed  views  may  be  required. 

5.  Develop  architecture  views 

Develop  architecture  views  in  accordance  with  DODAF  to  include  all  relevant 
activities  and  entities.  If  an  integrated  architecture  already  exists,  then  acquire  the 
required  architecture  views. 

6.  Develop  Modeling  Simulation  to  replicate  architecture 

a.  Select  Modeling  tool  best  suited  to  meet  evaluation  requirements  (i.e.  Excel 
spreadsheet  vs.  discrete  model  simulation  program) 
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b.  Model  structure  to  match  architecture  (OV-2,  OV-5) 

c.  Model  decision  logic  to  match  OV-6a. 

d.  Calculate  MOE"s  at  output  of  activities  as  functions  of  design  parameters 


7.  Evaluate  Model  Completeness 

Does  model  consider  all  relevant  aspects  (processes,  assumptions,  input  variables 
and  outputs,  MOE"s)  of  the  system/concept? 

a.  If  so,  continue  to  step  8. 

b.  If  model  not  complete,  return  to  step  3  with  the  following  considerations. 

i.  Determine  additional  architecture  view  and/or  level  of  abstraction 

required  to  achieve  traceability  between  system  and  the  missing 
aspect. 

ii.  Develop  required  additional  architecture 

iii.  Modify  model  to  include  additional  architecture  view. 

iv.  Re-evaluate  Step  7  until  model  captures  all  relevant  aspects  of  the 

concept. 

8.  Evaluate  model  for  MOE  results,  requirements  and  key  parameters 

a.  Once  the  model  is  complete,  evaluate  the  system’s  ability  to  meet  target 

metrics. 

b.  Vary  design  parameters  and  perform  sensitivity  analysis  to  identify  key 

parameters. 

c.  Compare  sensitivity  analysis  to  target  MOE’s  to  establish  requirements  and 

KPPs. 

d.  Identify  critical  performance  parameters  in  the  SV-7  Systems  Performance 

Parameters  Matrix. 

e.  Vary  system  design  and  design  parameters  to  evaluate  the  system’s  robustness 

and  its  rate  of  degradation. 
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Appendix  B:  ISHM  Architecture 


B.l  Integrated  System  Health  Management  Concept  of  Operations 
1.0  Purpose 

Integrated  System  Health  Management  (ISHM)  adds  a  centralized  health  management 
system  that  is  responsible  for  collecting  and  processing  health  status  information  from 
across  a  system  during  all  mission  phases.  ISHM  balances  data  flow  from  multiple  sub¬ 
systems  and  produces  the  information  necessary  to  identify  current  system  capabilities, 
provide  situational  awareness  to  mission  and  ground  operations,  and  quickly  identify 
contingencies  for  improved  vehicle  control  and  mission  decisions.  In  order  to  be 
effective,  ISHM  must  have  the  capability  to:  assess  vehicle  state;  reliably  detect, 
diagnose,  and  predict  failures  and  degraded  conditions;  derive  and  relay  accurate  vehicle 
health  status  to  the  ground  operations  crew,  maintainers,  and  the  on-flight  vehicle 
command  and  control  module.  These  capabilities  would  allow  the  operator  or  vehicle  to 
re-plan  the  mission,  reconfigure  flight  control  and  continue,  or  abort  as  necessary  in  real¬ 
time. 

2.0  Time  Horizon,  Assumptions,  and  Risks 

This  section  discusses  the  time  horizon  for  the  future  of  ISHM,  and  the  assumptions  and 
risks  overlaying  the  use  of  ISHM. 

2.1  Time  Horizon 

In  the  near  term  (0-10  yrs),  ISHM  is  envisioned  to  provide  condition-based  maintenance, 
remaining  life-quantification,  mission-readiness  decision  making,  and  improved  fault 
isolation  and  detection  to  the  operator. 

In  the  far  term  (10+  yrs),  as  systems  reach  for  true  autonomy,  ISHM  will  enable  an 
autonomous  vehicle  to  re -plan  its  own  mission  based  on  actual  system  health  and 
capabilities,  collaborate  with  other  autonomous  vehicles  to  ensure  mission  and  capability 
coverage,  and  define  its  own  operating  envelope. 

2.2  Assumptions 

(1)  The  ISHM  system  will  currently  have  no  command  or  control  over  the 

autonomous  vehicle;  it  only  provides  recommended  actions  for  flight  control 
and  ground  control. 

2.3  Risks 

(1)  If  ISHM  lacks  integrity,  the  false  alarm  rate  will  increase  the  probability  of 

unscheduled  maintenance  over  current  health  monitoring  systems. 

(2)  The  added  weight  of  an  ISHM  system  will  decrease  the  capability  of  the  system. 

(3)  The  added  cost  of  an  ISHM  could  outweigh  the  benefits  of  the  system. 
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(4)  If  ISHM  is  less  reliable  than  the  vehicle,  the  probability  of  unscheduled 
maintenance  will  increase  over  current  health  monitoring  systems. 

3.0  Description  of  the  Military  Challenge 

In  2010,  the  United  States  Air  Force  (USAF)  released  the  results  of  a  year-long  study 
highlighting  the  need  for  increasing  autonomy  in  modem  weapon  systems,  especially  in 
the  domain  of  unmanned  aerial  systems  (UAS).  The  study  identified  the  need  for  greater 
system  autonomy  as  the  “single  greatest  theme”  for  future  USAF  Science  and 
Technology  investments.  [Technology  Horizons,  2010]  Current  technology 
advancements  have  brought  the  USAF  to  a  state  of  flexible  autonomy,  which  involves 
dynamically  shifting  command  and  control  (C2)  from  autonomous  to  operator  based  on 
workload,  system  health,  and  the  perceived  intent  of  the  operator. 

One  of  the  key  attributes  sustaining  flexible  autonomy  is  the  ability  of  the  UAS  to  detect, 
isolate,  and  diagnose  system  health  problems  to  relay  back  to  ground  C2,  the  on-board 
flight  control  module,  and  maintainers  for  appropriate  action.  Current  flight  avionics 
architectures  may  include  lower  level  sub-system  health  monitoring  or  may  isolate  health 
monitoring  functions  to  a  black  box  configuration,  but  a  vehicle-wide  health  monitoring 
information  system  has  seldom  been  implemented. 

4.0  Synopsis 

ISHM  provides  the  basis  for  integrating  all  the  individual  system’s  health  management 
inputs  and  outputs  (I/O)  on  a  particular  vehicle  and  determines,  in  real-time,  the  vehicle’s 
health  status  and  mission  capabilities.  The  overall  desired  effect  of  an  ISHM  system 
would  be  an  increase  in  mission  success  rates,  driven  by  improved  operational 
availability,  increased  health  awareness,  faster  turnaround  times,  and  false  alarm 
avoidance.  In  order  to  perform  this  capability,  ISHM  must  provide  continuous  monitoring 
over  the  entirety  of  the  vehicle,  identify  that  a  fault  has  occurred,  pinpoint  the  fault 
mechanism  and  its  location,  assess  and  assign  a  level  of  health  to  the  vehicle,  and  relay 
selected  fault  data  to  the  ground  operator  for  action. 

5.0  Desired  Effects 

The  overall  desired  effect  of  an  ISHM  system  would  be  to  detect  and  isolate  either  a  real¬ 
time  fault  or  pre-cursers  to  a  fault,  determine  the  criticality  of  the  fault,  and  then  relay  the 
appropriate  information  back  to  ground  control  for  action.  Benefits  of  this  capability  are 
discussed  in  the  remaining  sub-sections. 

5.1  Effect  on  Scheduled  and  Unscheduled  Maintenance 

The  Air  Force  goal  for  prognostic  systems  such  as  ISHM  is  to  completely  eliminate 
traditional  aircraft  inspection  and  repair  patterns.  [Ross,  1999]  Currently,  a 
malfunctioning  unit  is  either  identified  in-flight  (based  off  an  alert  from  an  individual 
sensor)  or  identified  through  scheduled  inspections.  There  are  an  inherent  probability  of  a 
false  alarm  and  a  probability  of  fault  detection,  meaning  that  the  aircraft  could  be 
incorrectly  pulled  from  an  on-going  mission  or  could  continue  on  a  mission  with  an 
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unknown  fault  that  could  lead  to  system  failure.  The  integrated  aspect  of  ISHM  proposes 
to  severely  reduce  the  false  alarm  rate  and  increase  the  total  probability  of  detection,  as 
understanding  the  full  health  status  of  the  vehicle  can  identify  false  positives  and  identify 
if  a  fault  or  failure  has  occurred  down-stream.  For  example:  a  sensor  falsely  identifies  a 
valve  stuck  closed,  a  sensor  further  down  the  stream  indicates  a  normal  flow  rate  and  the 
system  has  not  lost  any  performance  aspects;  ISHM  would  therefore  not  report  this  as  a 
fault. 

With  the  continuous  monitoring  provided  by  ISHM,  pre-cursors  to  faults  can  also  be 
identified  and  an  Estimated  Time  to  Failures  of  the  component  or  total  system  will  be 
reported.  Additionally,  if  multiple  mission  data  is  stored,  every  time  a  fault  occurs,  the 
data  collected  by  ISHM  can  be  used  to  identify  new  indicators  or  pre-cursors  to  a  failure 
to  be  uploaded  into  the  diagnostic  and  prognostic  algorithms. 

The  overall  result  is  that  with  ISHM  implemented,  the  probability  of  unscheduled 
maintenance  should  be  minimized  and  scheduled  maintenance  intervals  can  be  relaxed  or 
removed.  Ideally  with  ISHM,  the  aircraft  would  replace  time-based  or  event-driven 
maintenance  with  a  condition-based  maintenance  system,  where  maintenance  is  only 
performed  based  on  objective  evidence  of  actual  or  predictable  failure  of  a  system  or  its 
components.  [OSAIDD,  1999] 

5.2  Decreased  Mean  Time  to  Repair 

Current  fault  detection  is  limited  to  identifying  the  occurrence  of  a  fault  and  an 
approximate  location,  meaning  that  fault  isolation  can  only  occur  after  the  aircraft  has 
landed.  There  is  also  a  probability  that  the  mechanic  can  even  correctly  identify  the 
failure  mode  once  it  lands.  With  ISHM,  both  fault  detection  and  isolation  are  performed 
in-flight,  within  a  specified  confidence  level,  and  the  appropriate  information  is  relayed 
to  the  maintenance  element.  This  gives  the  maintenance  element  time  to  pre -position  the 
necessary  maintenance  equipment  and  personnel  or  order  any  necessary  replacement 
parts,  severely  reducing  the  total  Mean  Time  to  Repair  (MTTR)  after  an  event. 

As  a  result  of  its  continuous  monitoring,  ISHM  would  also  reduce  maintenance  time 
during  scheduled  inspections.  Prognostics  would,  theoretically,  calculate  an  Estimated 
Time  to  Failure  (ETF)  for  each  component,  resulting  in  each  inspection  only  focusing  on 
those  systems  that  had  passed  an  ETF  threshold  in  that  time  interval.  Knowing  that  the 
specific  systems  to  be  inspected  ahead  of  time  would  again  allow  the  maintenance 
element  time  to  pre-position  the  necessary  maintenance  equipment  and  personnel  or  order 
any  necessary  replacement  parts.  ISHM  would  also  negate  the  current  use  of  time¬ 
intensive  Built-in  Test  (BIT)  units,  as  each  system  would  be  continuously  tested. 
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5.3  Operational  Availability  Improvement 

Based  on  the  decreased  downtime  in  unscheduled  maintenance  and  scheduled 
maintenance  from  ISHM,  the  Operational  Availability  for  each  aircraft  should  improve. 
Another  factor  affecting  Operational  Availability  is  mission  turn-around  time,  or  the  time 
from  when  the  aircraft  lands  to  when  it  is  ready  for  the  next  mission.  Without  ISHM, 
mission  turn-around  time  can  include  lengthy  BIT  tests  to  check  for  failures,  since  these 
tests  are  not  needed  with  continuous  monitoring  and  a  higher  confidence  in  fault 
detection  the  mission  turn-around  time  should  decrease,  increasing  Operational 
Availability.  Whether  measured  in  maintenance  downtime  or  a  reduction  in  hours 
required  for  testing  and  diagnostics,  etc.,  the  net  result  is  that  a  system  with  ISHM  will  be 
available  for  use  more  of  the  time. 

5.4  Increased  Mission  Success 

Having  situational  awareness  of  the  entire  health  state  of  the  vehicle  assists  ground 
operations  in  providing  full  mission  coverage.  If  a  UAS  autonomously  detects  a  fault  and 
due  to  the  fault  criticality  (for  example,  low  fuel  levels)  decides  to  re-task  to  a  closer 
trajectory,  ground  operations  can  re-task  other  UAS  vehicles  to  ensure  coverage  of  the 
priority  targets.  Without  ISHM,  a  fault  alert  would  generally  give  no  indication  of  the 
remaining  performance  capability  of  the  UAS,  leaving  ground  operations  to 
conservatively  scrap  that  particular  mission  set. 

An  additional  aspect  of  increased  situational  awareness  is  its  affect  on  UAS  flight  limits. 
Modern  autonomous  flight  control  systems  limit  the  vehicle  to  safe  operating  loads  and 
environments;  this  operating  envelope  is  pre-defined  and  very  conservative.  With  ISHM, 
the  flight  envelope  can  theoretically  be  expanded  and  defined  by  the  design  criteria  for 
the  vehicle.  Health  data  would  then  be  used  to  restrict  the  envelope  to  a  prescribed  level 
in  the  event  of  a  detected  fault.  This  would  increase  the  operational  capability  of  the 
vehicle,  allowing  for  larger  mission  sets.  Improved  situational  awareness  combined  with 
the  theoretical  improved  Operational  Availability  would  greatly  improve  the  rate  of 
mission  success. 

5.5  Cost  Savings 

The  previous  benefits  all  have  some  measure  of  cost  savings  attached  to  them.  Having  a 
lower  total  maintenance  downtime,  due  to  decreases  in  scheduled  maintenance  and  a 
lower  probability  of  unscheduled  maintenance,  leads  to  a  lower  personnel  cost  and  even 
an  option  of  having  less  maintenance  personnel  needed.  Fewer  maintenance  actions  also 
indicate  a  potential  reduction  in  spares  and  supply  costs.  The  on-board  test  diagnostics 
provided  by  ISHM  would  also  theoretically  replace  some  ground  test  equipment,  as  it 
would  become  redundant. 
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6.0  Necessary  Capabilities 

The  capabilities  necessary  for  ISHM  to  be  effective  are  appropriate  data  and  information 
management,  fault  detection  and  isolation,  the  ability  to  assess  the  health  status  of  the 
UAS,  and  communication,  both  internal  and  external  to  the  system. 

6.1  Data  Management 

The  ISHM  system  must  provide  continuous  monitoring  over  the  entirety  of  the  vehicle. 
Sensors  are  placed  in  critical  locations  in  order  to  feed  information  on  the  state  of  the 
system.  Sensors  can  be  conventional,  measuring  temperature,  speed,  and  flow  rate,  or 
specifically  tailored  to  health  management  applications,  such  as  strain  gauges,  ultrasonic 
sensors,  or  proximity  devices. 

Data  Management  also  includes  parameter  sets,  vehicle  configuration,  and  a  data  store 
with  a  list  of  safe  states  associated  with  known  fault  events  and  mitigation  steps.  Current 
mission  sensor  data  and  event  recording  can  either  be  kept  in  an  on-board  data  storage 
system  sent  to  ground  as  required,  or  continuously  streamed  to  ground  control. 

6.2  Fault  Detection 

The  sensor  data  is  then  processed  to  remove  any  artifacts  or  noises  and  manipulated  to 
extract  fault  features  (either  current  or  pre-cursers)  and  provide  a  comprehensive  system 
picture.  Fault  Detection  combines  diagnostic  information  with  historical  data  (prognostic 
reasoning)  to  generate  an  estimation  of  failure  times.  These  fault  indications  are  then 
sorted,  prioritized,  and  distributed  to  insure  action  within  time  to  criticality.  Algorithms 
developed  for  diagnostic  and  prognostic  calculations  are  generally  based  on  mathematical 
models  (e.g.  Hamilton  dynamic,  Lagrangian  dynamic,  approximation  methods),  or 
pattern  recognition  (e.g.  fuzzy-logic,  statistical/regression  methods,  neural  network 
clustering). 

6.3  Fault  Isolation 

After  identifying  that  a  fault  has  occurred,  ISHM  must  pinpoint  the  fault  mechanism  (i.e. 
the  specific  cause  of  failure)  and  its  location.  If  not  identifiable  through  prognostic 
reasoning,  common  fault  mechanisms  for  that  location  can  be  identified  using  historical 
failure  data. 

6.4  Health  State  Assessment 

ISHM  must  have  the  capability  to  assess  and  assign  levels  of  health  to  the  vehicle.  This  is 
achieved  by  calculating  the  remaining  vehicle  capabilities  based  on  a  capability  model 
and  the  current  fault  state  of  the  system.  A  notional  capability  model  is  hierarchically 
based,  where  the  higher- level  capability  is  computed  using  the  values  of  the  lower- level 
capabilities  and  a  mathematical  expression.  Faults  are  quantified  at  the  lowest  level  with 
system-level  capability  computations  that  orient  this  data  with  mission  requirements  to 
determine  effects  on  the  vehicle. 
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6.5  Select  Mitigation  Procedures 

The  ISHM  system  will  provide  mitigation  procedures  in  the  event  of  a  known  fault  for 
the  on-board  flight  control  to  act  on  if  necessary.  In  order  perform  this  capability,  ISHM 
will  a)  examine  the  available  resources  to  determine  any  performance  limitations  and  to 
estimate  the  time  to  criticality;  b)  confirm  the  diagnosed  event  and  declare  it  to  be  a  valid 
vehicle  event  with  a  high  confidence  level;  c)  access  the  fault  data  store  for  the 
appropriate  safe  state  and  the  feasible  step  alternatives;  before  d)  selecting  the  action 
steps  that  allow  completion  within  the  criticality  time  and  performance  limitations.  These 
action  procedures  will  then  be  sent  to  the  on-board  flight  control  and  to  ground  control. 
Since  ISHM  is  deterministic  and  operates  only  on  known  faults  and  known  mitigations, 
any  unknown  fault  will  immediately  be  assigned  a  critical  level  of  health  and  the  aircraft 
will  automatically  return  to  base. 

6.6  Communication 

The  ISHM  must  be  able  to  send  and  receive  messages  internally  and  externally  to  the 
vehicle. 

7.0  Enabling  Capabilities 

A  formal  Failure  Mode,  Effect,  and  Criticality  Analysis  (FMECA)  must  be  performed  on 
the  vehicle  prior  to  ISHM  being  implemented.  This  is  an  iterative  process  that  identifies 
failure  modes,  assesses  their  probabilities  of  occurrence  and  their  effects  on  the  system, 
isolating  their  causes,  and  determining  corrective  action  or  preventative  measures.  The 
results  of  the  FMECA  should  identify  critical  sub-systems  or  components  where  sensors 
need  to  be  applied,  guide  the  diagnostic  and  prognostic  algorithm  creation,  and  assign 
criticality  to  failure  modes  for  health  assessment  purposes. 

8.0  Sequenced  Actions 

There  are  three  main  use  cases  for  ISHM:  no  faults  occur,  a  fault  event  occurs  real-time, 
and  pre-cursors  to  a  fault  are  identified.  See  Appendix  11.1  for  key  nomenclature 
definitions. 

8.1  Nominal  Operations 

The  ISHM  system  will  be  continuously  monitoring  the  health  state  of  the  UAS  and  will 
communicate  either  continuously  or  on  set  intervals  (barring  a  fault  event)  the  health 
status  of  the  UAS.  ISHM  will  also  be  continuously  calculating  an  Estimated  Time  to 
Failures  for  every  monitored  component. 

8.2  Real-Time  Fault  Event 

Once  a  failure  occurs,  the  following  actions  should  take  place: 

(1)  ISHM  locates  the  fault  and  identifies  the  failure  mode 

(2)  ISHM  assigns  a  criticality  to  the  fault  mode  and  adjusts  the  vehicle’s  health 
status  to  the  appropriate  level 

(3)  ISHM  evaluates  the  new  capability  of  the  vehicle 

(4)  ISHM  selects  appropriate,  deterministic,  mitigation  action  procedures, 
correlating  them  with  mission  and  vehicle  inhibits. 
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(5)  ISHM  sends  the  action  procedures  to  the  on-board  flight  control,  and  alerts  the 
Ground  Operator  and  the  Maintenance  element 

a.  The  on-board  flight  control  can: 

(i)  Continue  on  current  trajectory  (ignore  ISHM) 

(ii)  Use  recommendations  to  autonomously  reconfigure  and/or  reshape  the 
current  trajectory 

b.  The  Ground  Operator,  as  appropriate  and  in  accordance  with  the  criticality 
of  the  event,  can: 

(i)  Override  the  on-board  flight  control  decision  and  re-task  within  its  new 
capability 

(ii)  Defer  to  the  autonomous  on-board  flight  control  decision 

c.  The  Maintenance  element  executes  maintenance  actions  as  appropriate 

8.3  Pre-Cursor  to  Fault  is  Detected 

When  a  pre-cursor  to  a  fault  is  detected,  the  following  actions  should  take  place: 

(1)  ISHM  locates  the  affected  component  and  identifies  the  impending  failure 
mode 

(2)  ISHM  calculates  an  Estimated  Time  to  System  (or  Component)  Failure 

(3)  ISHM  assigns  a  criticality  to  the  fault  mode  and  adjusts  the  vehicle’s  health 
status  to  the  appropriate  level 

(4)  ISHM  evaluates  the  new  capability  of  the  vehicle 

(5)  ISHM  selects  appropriate,  deterministic,  mitigation  action  procedures, 
correlating  them  with  mission  and  vehicle  inhibits. 

(6)  ISHM  sends  the  action  procedures  to  the  on-board  flight  control,  and  alerts  the 
Ground  Operator  and  the  Maintenance  element 

a.  The  on-board  flight  control  can: 

(i)  Continue  on  current  trajectory  (ignore  ISHM) 

(ii)  Use  recommendations  to  autonomously  reconfigure  and/or  reshape  the 
current  trajectory 

b.  The  Ground  Operator,  as  appropriate  and  in  accordance  with  the  criticality 
of  the  event,  can: 

(i)  Override  the  on-board  flight  control  decision  and  re-task  within  its  new 
capability 

(ii)  Defer  to  the  autonomous  on-board  flight  control  decision 

c.  The  Maintenance  element  executes  maintenance  actions  as  appropriate 

9.0  Command  Relationships 

ISHM  will  have  no  command  and  control  over  the  UAS  at  this  time.  The  ISHM  system 
will  need  to  communicate  with  the  following  systems/subsystems: 

9.1  Ground  Control 

Ground  systems  are  normally  treated  as  separate  systems,  and  their  relationship  to  the 
vehicle  has  typically  been  one  of  controller  and  operator;  in  this  case,  ground  is 
hierarchically  superior  to  the  vehicle  and  commands  it.  For  events  that  happen  during  a 
mission,  the  ground  will  take  control  to  determine  the  needed  actions  and  then  send  the 

98 


commands  to  the  vehicle  for  execution.  The  vehicle  interfaces  with  ground  control  to 
capture,  analyze,  and  preserve  vehicle  health  data. 

Vehicle  control  transitions  between  ground  and  on-board  depending  on  mission  phase 
and  particular  event  conditions: 

•  Before  Launch 

o  Ground  is  master 

o  Control  transitions  to  vehicle  during  launch  sequence 

•  During  Flight 

o  Vehicle  is  master  (autonomous) 
o  Ground  monitors  via  downlink  telemetry 
o  Ground  takes  control  when  appropriate 

•  Post  Landing 

o  Ground  is  master  (after  auto-safing) 

9.2  Maintenance  and  Logistics 

Maintenance  and  Logistics  can  be  considered  part  of  ground  control  (under  the 
overarching  domain  of  “Operations  Control  Center”)  or  a  separate  system  entirely.  Their 
relationship  to  the  vehicle  is  either  reactionary  or  scheduled  and  does  not  consist  of  a 
hierarchical  relationship. 

Interactions: 

•  Scheduled  Maintenance:  Based  on  flight  hours  and  is  performed  at  either  the 
base-level  or  at  a  depot.  Collected  historical  data  from  ISHM  monitoring  can  be 
used  to  highlight  components  that  need  to  be  inspected. 

•  Unscheduled  Maintenance:  Initiated  when  a  fault  has  been  discovered.  Once  the 
ISHM  has  detected  an  anomaly,  the  appropriate  data  is  sent  to  Maintenance  and 
Logistic  for  action. 

•  Post  Mission:  Degradation  and  non-critical  fault  information  are  sent  to 
Maintenance  and  Logistics  to  improve  vehicle  turn-around  time. 

9.3  Vehicle  Systems 

ISHM  collects  status  and  event  snapshots  from  the  vehicle  subsystems  and  processes  the 
information  using  various  health  algorithms  and  reasoning  capabilities. 

9.4  On-Board  Flight  Control 

The  on-board  flight  control  receives  command  to  execute  an  action  from  ISHM  generated 
by  either  ISHM  and/or  ground  C2.  The  autonomous  on-board  flight  control  will 
decompose  these  decisions  and  action  lists  into  a  set  of  commands  and  send  them  to  the 
appropriate  systems  for  execution.  On-board  flight  control  schedules  these  tasks 
accordingly  in  order  to  complete  in  the  prescribed  time. 

As  a  vehicle  system,  on-board  flight  control  health  status,  events,  time,  and  mission 
information  are  continuously  sent  to  ISHM.  ISHM  in  turn  continuously  provides  the 
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vehicle  system  health  assessments,  vehicle  capability,  and  mitigation  actions 
predetermined  for  particular  anomalies. 
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B.2  Architecture  Concept  of  Operations 


1.0  Purpose 

The  focus  of  this  research  is  to  quantify  the  mission-related  benefits  of  ISHM  by 
constructing  architecture  for  analysis  to  compare  against  current  autonomous  vehicle 
capabilities,  and  to  provide  a  general  baseline  model  that  can  be  implemented  over  any 
current  or  future  autonomous  vehicle.  The  architecture  capabilities  will  include  the  ability 
to  analyze  the  causal  relationship  of  ISHM  performance  metrics  (to  include  the 
performance  of  the  processor,  and  the  performance  and  reliability  of  the  monitoring 
sensors)  to  mission  performance.  The  architecture  is  aimed  at  primarily  answering  the 
following  questions: 

(1)  What  are  the  potential  impacts  to  ground  control  stations  and  users? 

(2)  What  are  the  potential  impacts  on  current  maintenance  practices? 

(3)  What  are  the  performance  characteristics  necessary  for  ISHM  to  effect 

mission  success? 

The  architecture  should  also  contribute  to  answering  these  secondary  questions: 

(1)  How  should  the  ISHM  data  be  presented? 

(2)  Is  ISHM  cost  effective? 

2.0  Time  Horizon,  Assumptions,  and  Risks 

This  section  discusses  the  time  horizon  for  the  architecture,  the  assumptions  overlaying 
the  architecture,  and  the  risks  inherent  in  using  this  architecture  and  analysis  tool. 

2.1  Time  Horizon 

The  architecture  and  preliminary  analysis  should  be  built  and  completed  by  December  of 
2012  with  the  project  out-brief  scheduled  for  March  of  2013.  Additional  interim  gates 
will  be  established  as  the  project  progresses. 

2.2  Assumptions 

At  this  point  in  ISHM  development: 

(1)  This  architecture  does  not  analyze  ISHM  past  the  system  level;  however,  the 

architecture  can  be  easily  modified  to  include  components  and  subsystems 
that  are  of  value  to  the  researcher. 

(2)  At  this  point  in  ISHM  development,  the  ISHM  system  has  no  command  or 

control  over  the  autonomous  vehicle.  ISHM  in  its  current  technology  state 
only  provides  recommended  actions  based  on  the  type  of  fault  it  detects.  The 
vehicle’s  autonomous  management  system  will  prioritize  actions  as 
necessary  or,  if  there  is  time,  a  ground-based  ISHM  team  can  overrule  or 
direct  mitigation  actions  as  necessary. 

2.3  Risks 

(1)  If  the  selected  metrics  for  analysis  either  do  not  accurately  describe  ISHM  or 
are  not  independent  of  each  other,  the  optimization  process  will  give  an 
analysis  that  does  not  appropriately  model  the  system. 
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3.0  Description  of  the  Military  Challenge 

ISHM  is  currently  being  considered  for  implementation  in  unmanned  aerial  systems 
(UAS)  for  the  United  States  Air  Force.  Before  the  USAF  can  move  forward,  the  effects 
of  ISHM  both  on-ground  and  in-flight  need  to  be  understood  and  evaluated.  This  project 
intends  to  perform  an  in-depth  analysis  on  the  advantages  and  disadvantages  of  adding  an 
ISHM  system  to  general  or  specific  UAS  through  the  use  of  DODAF  architecture  and 
optimization  processes. 

4.0  Synopsis 

This  concept  intends  to  perform  an  in-depth  analysis  on  the  advantages  and  disadvantages 
of  adding  an  ISHM  system  to  a  general  or  specific  UAS  through  the  use  of  architecture. 
The  architecture  will  have  the  capability  to  optimize  a  given  ISHM’s  fault  detection  rate, 
fault  isolation  coverage  rate,  false  alarm  rate,  and  calculate  the  mean  time  to  repair,  mean 
time  between  failure,  probability  of  scheduled  maintenance,  probability  of  unscheduled 
maintenance,  and  turn-around  time  for  a  UAS  with  ISHM.  To  use  the  architecture  and 
analysis  tool,  the  user  will  select  desired  metrics,  modify  the  system’s  objectives  and 
constraints,  input  the  results  of  the  Failure  Mode,  Effect,  and  Criticality  Analysis, 
integrate  the  ISHM  architecture  into  the  overall  system  architecture,  execute  the  analysis, 
and  then  perform  sensitivity  analysis  on  the  results. 

5.0  Desired  Effects 

This  architecture  should  provide  a  general  baseline  model  that  can  be  implemented  over 
any  autonomous  vehicle  and  to  quantify  the  effect  of  ISHM  on  the  operational 
availability  and  mission  success  rate  by  comparing  them  to  current  autonomous  vehicle 
capabilities.  The  architecture  should  have  the  capability  to  optimize  the  ISHM  fault 
detection  rate,  fault  isolation  coverage  rate,  false  alarm  rate,  and  the  expected  weight  of 
the  ISHM.  The  architecture  should  have  the  capability  to  use  those  optimized  rates  to 
calculate  these  metrics  for  a  UAS  with  ISHM:  mean  time  between  failures  (MTBF), 
probability  of  scheduled  maintenance,  probability  of  unscheduled  maintenance, 
operational  availability,  and  the  probability  of  mission  success. 

6.0  Necessary  Capabilities 

The  capabilities  necessary  to  use  the  ISHM  architecture  are  that  the  architecture  is 
flexible,  supports  analysis  and  optimization,  and  is  easy  to  use. 

6.1  Flexible 

The  architecture  should  have  the  capability  to  be  modified  to  fit  any  UAS  baseline 
architecture.  This  architecture  should  also  have  the  capability  to  be  expanded  on  for 
future  generations  of  UAS  and  ISHM  technologies. 

6.2  Analysis  and  Optimization 

This  architecture  should  have  the  capability  to  support  an  evaluation  model.  The 
particular  modeling  or  analytical  tools  (such  as  spreadsheets  or  discrete  event  simulation, 
or  through  a  simulation  software  product  such  as  ARENA)  can  be  chosen  by  the  user. 
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6.3  Ease  of  Use 

The  architecture  and  analysis  process  should  be  straightforward  and  clear.  Any  user  that 
has  some  prior  knowledge  of  ISHM  and  DoDAF  architecture  should  have  the  ability  to 
understand  the  architecture  and  perform  some  level  of  modification  as  appropriate  and 
appreciate  its  use  as  an  analysis  tool. 

7.0  Enabling  Capabilities 

In  order  to  operate  the  architecture,  a  Failure  Mode,  Effect,  and  Criticality  Analysis  must 
be  performed,  and  the  overall  architecture  of  the  system  must  be  built. 

7.1  Failure  Mode,  Effect,  and  Criticality  Analysis 

A  formal  Failure  Mode,  Effect,  and  Criticality  Analysis  (FMECA)  must  be  performed  on 
the  UAS  prior  to  using  this  ISHM  architecture.  This  is  an  iterative  process  that  identifies 
failure  modes,  assesses  their  probabilities  of  occurrence  and  their  effects  on  the  system, 
isolating  their  causes,  and  determining  corrective  action  or  preventative  measures.  The 
results  of  the  FMECA  should  identify  critical  sub-systems  or  components  where  sensors 
need  to  be  applied,  guide  the  diagnostic  and  prognostic  algorithm  creation,  and  assign 
criticality  to  failure  modes  for  health  assessment  purposes. 

7.2  UAS  Architecture 

The  architecture  for  the  overall  system  for  which  ISHM  is  going  to  be  analyzed  should  be 
should  be  built  prior  to  using  this  lower  level  ISHM  architecture;  the  intent  of  the  lower 
level  ISHM  architecture  is  to  be  integrated  into  the  overall  system  architecture.  The 
following  metrics  from  the  baseline  vehicle  should  be  collected  for  use  in  the 
architecture:  Mean  Time  to  Repair  (MTTR),  MTBF,  rate  of  scheduled  maintenance, 
probability  of  unscheduled  maintenance,  operational  availability,  and  the  probability  of 
mission  success. 

8.0  Sequenced  Actions 

To  execute  the  architecture,  take  the  following  steps: 

(1)  Select  Metrics:  Select  the  metrics  that  the  user  is  interested  in  for  analysis. 

(2)  Modify  Objectives  and  Constraints:  Modify  the  selected  objectives  and 
constraints  to  reflect  the  metrics  that  the  user  is  interested  in. 

(3)  Input  results  of  FMECA:  Enter  in  the  failure  data  for  each  subsystem  as  a 
parametric  distribution.  Assign  criticality  to  identified  failure  modes. 

(4)  Modify  the  Architecture:  Integrate  the  ISHM  architecture  into  the  overall 
system  architecture.  Use  the  results  of  the  FMECA  to  highlight  the  critical 
systems  that  need  to  be  monitored  and  determine  the  number  of  sensors  to  be 
implemented  in  each  system. 

(5)  Execute  Analysis 

(6)  Perform  Sensitivity  Analysis. 
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B.3  AY-1 


1.0  Architectural  Description 

Previous  generation  health  monitoring  technology  was  typically  local  to  a  given 
subsystem;  the  next  generation,  Integrated  System  Health  Management  (ISHM),  adds  a 
centralized  health  management  system  to  a  typical  flight  avionics  configuration.  It  is 
responsible  for  collecting  and  processing  vehicle  health  status  information  from  across 
the  vehicle  during  all  mission  phases.  As  a  consequence  it  will  enhance  the  ability  to 
make  on-board  decisions,  thus  migrating  strict  ground  control  to  shared  vehicle 
autonomy.  ISHM  performs  health  management  at  the  vehicle-  and  mission-level  from 
events  and  diagnostics  gathered  at  the  subsystem  level. 

ISHM  is  currently  being  considered  for  implementation  in  unmanned  aerial  systems 
(UAS)  for  the  United  States  Air  Force  (USAF).  Before  the  USAF  can  move  forward,  the 
effects  of  ISHM  both  on-ground  and  in-flight  need  to  be  understood  and  evaluated.  The 
focus  of  this  architecture  is  to  quantify  the  mission-related  benefits  of  ISHM  by 
constructing  architecture  for  analysis  to  compare  against  current  autonomous  vehicle 
capabilities,  and  to  provide  a  general  baseline  model  that  can  be  implemented  over  any 
current  or  future  autonomous  vehicle.  The  architecture  capabilities  will  include  the  ability 
to  analyze  the  causal  relationship  of  ISHM  performance  metrics  (to  include  the 
performance  of  the  processor,  and  the  performance  and  reliability  of  the  monitoring 
sensors)  to  mission  performance. 

2.0  Scope 

The  architecture  is  aimed  at  primarily  answering  the  following  questions: 

(1)  What  are  the  potential  impacts  to  ground  control  stations  and  users? 

(2)  What  are  the  potential  impacts  on  current  maintenance  practices? 

(3)  What  are  the  performance  characteristics  necessary  for  ISHM  to  effect 

mission  success? 

The  architecture  should  also  contribute  to  answering  these  secondary  questions: 

(1)  How  should  ISHM  data  be  presented  to  be  effective?  How  will  the 
presentation  change  in  regards  to  the  different  users  of  ISHM  (operators, 
maintenance,  etc.)? 

(2)  Is  ISHM  cost  effective? 

2.1  Architectural  Views  and  Products  Contained 

This  architecture  contains  Operational  and  System  views. 

The  Operational  views  include  a  High  Level  Operational  Concept  Graphic  (OV-l)  to 
graphically  describe  the  operational  concept,  an  Operational  Resource  Flow  Description 
(OV-2)  to  describe  the  resource  flows  exchanged  between  operational  activities,  and 
Operational  Activity  Models  (OV-5a  and  OV-5b)  to  describe  the  relationships,  inputs, 
and  outputs  between  operational  activities.  These  Operational  views  model  the  static 
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structure  of  the  architectural  elements  and  their  relationships.  An  additional  Operational 
view  that  describes  dynamic  behavior  is  the  Operational  Rules  Model  (OV-6a),  which 
defines  operational  procedures  and  constraints. 

The  planned  system  view  is  the  Systems  Interface  Description  (SV-1),  which  identifies 
systems,  system  items,  and  their  interconnections. 

2.2  Project  Timeline 

The  architecture  and  preliminary  analysis  should  be  built  and  completed  by  December  of 
2012  with  the  project  out-brief  scheduled  for  March  of  2013.  Additional  interim  gates 
will  be  established  as  the  project  progresses. 

3.0  Purpose  and  Perspective 

The  purpose  of  the  architecture  is  to  provide  a  general  baseline  model  that  can  be 
implemented  over  any  autonomous  vehicle  and  to  quantify  the  effect  of  ISHM  on  the 
operational  availability  and  mission  success  rate  by  comparing  them  to  current 
autonomous  vehicle  capabilities.  The  architecture  should  have  the  capability  to  optimize 
the  ISHM  fault  detection  rate,  fault  isolation  coverage  rate,  false  alarm  rate,  and  the 
expected  weight  of  the  ISHM.  The  architecture  should  have  the  capability  to  use  those 
optimized  rates  to  calculate  these  metrics  for  a  UAS  with  ISHM:  mean  time  between 
failures  (MTBF),  probability  of  scheduled  maintenance,  probability  of  unscheduled 
maintenance,  operational  availability,  and  the  probability  of  mission  success. 

4.0  Tools  and  File  Formats  Used 

The  architecture  will  be  built  in  Enterprise  Architect  v8.0  (student)  and  presented  in  three 
formats:  Word  documents,  HTML  reports,  and  XML  data  files. 

5.0  Assumptions  and  Constraints 

This  section  includes  assumptions  and  constraints  needed  to  understand  the  architecture 
and  its  intended  usage. 

5.1  Assumptions 

(1)  Cost  will  not  be  evaluated 

(2)  At  this  point  in  ISHM  development,  the  ISHM  system  has  no  command  or 

control  over  the  autonomous  vehicle.  ISHM  in  its  current  technology  state 
only  provides  recommended  actions  based  on  the  type  of  fault  it  detects.  The 
vehicle’s  autonomous  management  system  will  prioritize  actions  as 
necessary  or,  if  there  is  time,  a  ground-based  ISHM  team  can  overrule  or 
direct  mitigation  actions  as  necessary. 

5.2  Constraints 

In  order  to  make  a  general  baseline  model,  it  is  not  possible  to  represent  every  possible 
aspect  of  ISHM.  Therefore  this  architecture  will  not  analyze  ISHM  past  the  system  level; 
however,  the  architecture  can  be  easily  modified  to  include  components  and  subsystems 
that  are  of  value  to  the  researcher. 
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6.0  Supporting  Analysis 

Representative  data  for  a  UAS  will  be  fed  into  the  model  and  Design  of  Experiments 
(DOE)  techniques  will  be  used  to  determine  situations  where  ISHM  can  be  effectively 
used.  The  response  for  this  analysis  is  the  difference  between  the  number  of  successful 
missions  calculated  for  a  system  without  ISHM  (i.e.  using  current  health  management 
techniques)  and  a  system  with  ISHM.  The  intent  is  to  explore  the  response  surface  where 
this  difference  is  maximized,  which  coincides  with  the  operational  area  where  ISHM 
would  be  most  beneficial. 

7.0  Findings 

A  response  surface  was  modeled  for  a  UAS  with  an  expected  lifetime  of  10,000  hours, 
maintenance  interval  of  1,000  hours,  and  average  mission  length  of  10  hours.  The  final 
model  equation  mapping  this  surface  was  determined  to  be: 

y  =  (-5.861  +  0.0150  +  17.322PD  -  67 369 P (Fault  Cat  III )  +  0.007c 

+  501.715(P(Fait/t  Cat  III )  -  0.191)2  -  0.237(0  -  850)(PD  -  0.652) 

-  0.0001(0  -  850)2  +  0.391)  -  0.415(P(Fait/t  Cat  III )  -  0.191)(c 

-  300))2 

where  y  is  the  difference  between  the  number  of  successful  missions  calculated  for  a 
system  with  ISHM  and  the  same  system  without  ISHM  (i.e.  using  current  health 
management  techniques) 

Contour  plots  for  the  response  surface  near  this  point  are  provided  in  the  figures  below. 
The  statistical  analysis  determined  that  a  response  greater  than  4.726  indicated  a 
statistically  significant  difference  in  mission  success  rates.  The  shaded  regions  on  the 
contour  plots  indicate  areas  where  ISHM  is  not  beneficial.  If  the  factors  fall  anywhere 
outside  of  this  region,  ISHM  should  be  investigated  as  a  beneficial  addition  to  the 
existing  UAS  in  terms  of  mission  success  rates. 
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B.4  OV-1 
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B.5  OV-2 


«0V-2»  class  OV-2  [OV-2] 
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B.6  OV-5a 


B.7  OV-5b 
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«OV-5»  act  OV-5  [0V-5b  Fault  During  Mission] 
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«0V-5»  act  OV-5  [0V-5b  Lifetime  Operations] 
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B.8  OV-6a 
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B.9  SV-1 


«SV-1»  composite  structure  SV-1  [SV-1] 
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Appendix  C:  Analytic  Model  Code 


Dim  ConfMatrix(6,  6)  As  Double 

Dim  RandCat  As  Double,  TrueCat  As  Single,  Detect  As  Single 

Dim  SysCatl,  SysCat2,  SysCat3,  SysCat4,  SysCat5,  SysParaml,  SysParam2 

Dim  NumFalseAlarms,  NumFaults,  NumFaultsDetected,  NumlncorrectDeclared 

Dim  ProbAlarm,  ProbDetect,  ProbDiagnostic,  MissionSuccessOld,  MissionSuccessNew 

Dim  MissionLength,  MaintLength,  MonteCarloNum,  Lifetime,  NumMissions 

Dim  Results()  As  Integer,  SumTotal(8)  As  Integer 

Sub  UserForm_Start() 

UserForml  .Show 

End  Sub 

Sub  MonteCarloSim(flag  As  Boolean) 

If  flag  =  False  Then 

With  Worksheets("HiddenCM") 

For  i  =  1  To  6 
For  j  =  1  To  6 

ConfMatrix(i,  j)  =  .Cells(7  +  i,  2  +  j).' Value 
Next  j 
Next  i 
End  With 
End  If 

W  orksheets(  "Calculations ")  .Activate 

ReDim  Results(MonteCarloNum,  8) 

For  i  =  1  To  8 
SumTotal(i)  =  0 
Next  i 

Run  simulation 

With  Worksheets("Calculations") 

Application.Goto  .Range("Al:P38") 

ActiveWindow.Zoom  =  True 
.Cells(5,  13). Select 

'Populate  Calculation  page 
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.Cells(4,  2). Value  =  ProbDetect 
.Cells(5,  2). Value  =  Prob Alarm 
.Cells(6,  2). Value  =  ProbDiagnostic 
.Cells(4,  5). Value  =  SysCatl 
.Cells(5,  5). Value  =  SysCat2 
.Cells(6,  5). Value  =  SysCat3 
.Cells(7,  5). Value  =  SysCat4 
.Cells(8,  5). Value  =  SysCat5 
.Cells(3,  ll).Value  =  MissionLength 
.Cells(4,  ll).Value  =  NumMissions 
.Cells(5,  ll).Value  =  MaintLength 
.Cells(6,  ll).Value  =  Lifetime 
.Cells(7,  ll).Value  =  MonteCarloNum 

MaintNum  =  Int(Lifetime  /  MaintLength) 

For  i  =  1  To  MonteCarloNum 

NumFalseAlarms  =  0 
NumFaultsDetected  =  0 
NumFaults  =  0 
NumlncorrectDeclared  =  0 
NumSuccessMsns_01d  =  0 
NumMaint_01d  =  0 
NumSuccessMsns_New  =  0 
NumMaint_New  =  0 

For  j  =  1  To  MaintNum 

'System  Distribution 

If  UserForml.SystemDist  =  "Normal"  Then 

tempi  =  Application.WorksheetFunction.NormInv(Rnd(),  SysParaml, 
SysParam2) 

Elself  UserForml.SystemDist  =  "Lognormal"  Then 

tempi  =  Application.WorksheetFunction.LogInv(Rnd(),  SysParaml, 
SysParam2) 

Elself  UserForml.SystemDist  =  "Weibull"  Then 

tempi  =  SysParaml  *  (-Log(l  -  Rnd()))  A  (1  /  SysParam2) 

Else  'Gamma' 

tempi  =  Application.WorksheetFunction.GammaInv(Rnd(), 
SysParaml,  SysParam2) 

End  If 

tempHours  =  0 
NumFaults_temp  =  0 
NumFalseAlarms_temp  =  0 
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NumFaultsDetected_temp  =  0 
NumIncDecl_temp  =  0 
NumSuccessMsns_01d_temp  =  0 
NumMaint_01d_temp  =  0 
NumSuccessMsns_New_temp  =  0 
NumMaint_New_temp  =  0 
flagX  =  False 
flagY  =  False 

For  k  =  1  To  NumMissions 

tempHours  =  tempHours  +  MissionLength 
tempRow  =  Range("A13").End(xlDown).Offset(l).Row 
.Cells(tempRow,  1)  =  tempHours 

'Fault  Occured 
If  tempi  <  tempHours  Then 
Fail  =  1 

NumFaults_temp  =  NumFaults_temp  +  1 
Else 
Fail  =  0 
End  If 

.Cells(tempRow,  2)  =  Fail 

Randl  =  Rnd() 

Rand2  =  Rnd() 

'Detection  Prob 
If  Fail  =  1  Then 

If  Randl  <  ProbDetect  Then 
Detect  =  1 
Else 

Detect  =  0 
End  If 
Else 

If  Rand2  <  Prob  Alarm  Then 
Detect  =  1 
Else 

Detect  =  0 
End  If 
End  If 

.Cells(tempRow,  3)  =  Detect 

'False  Alarms?  Detected  Failure? 

If  Fail  =  0  And  Detect  =  1  Then 

N  u  m  Fal  sc  A 1  arm  s_tcm  p  =  NumFalseAlarms_temp  +  1 
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End  If 

If  Fail  =  1  And  Detect  =  1  Then 

NumFaultsDctcctcdtcmp  =  NumFaultsDetected_temp  +  1 
End  If 

RandCat  =  Rnd() 

True  Fault  Category 
If  RandCat  <  SysCatl  Then 
TrueCat  =  1  *  Fail 
Elself  RandCat  <  SysCat2  Then 
TrueCat  =  2  *  Fail 
Elself  RandCat  <  SysCat3  Then 
TrueCat  =  3  *  Fail 
Elself  RandCat  <  SysCat4  Then 
TrueCat  =  4  *  Fail 
Else 

TrueCat  =  5  *  Fail 
End  If 

.Cells(tempRow,  4)  =  TrueCat 
Declared  Fault  Category 

DetectCat  =  DeclareMatrix(RandCat,  TrueCat,  Detect) 

.Cells(tempRow,  4)  =  TrueCat 
.Cells(tempRow,  5)  =  DetectCat 

'Incorrectly  Declared? 

If  TrueCat  <>  DetectCat  Then 

NumIncDecl_temp  =  NumIncDecl_temp  +  1 
End  If 

'Mission  Success  Calculations 

If  DetectCat  >=  1  Or  TrueCat  >=  3  Or  flagX  =  True  Then 
SuccessMsn  =  0 
Else 

SuccessMsn  =  1 

NumSuccessMsns_01d_temp  =  NumSuccessMsns_01d_temp  +  1 
End  If 

.Cells(tempRow,  7)  =  SuccessMsn 

If  DetectCat  >=  4  Or  (TrueCat  >=  3  And  DetectCat  <=  2)  Or  _ 
(TrueCat  >=  4  And  DetectCat  =  3)  Or  TrueCat  =  5  _ 

Or  flagY  =  True  Then 
SuccessMsn2  =  0 
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Else 

SuccessMsn2  =  1 

NumSuccessMsns_New_temp  =  NumSuccessMsns_New_temp  +  1 
End  If 

.Cells(tempRow,  9)  =  SuccessMsn2 

'Maintenance  Required? 

If  DetectCat  >=  1  And  DetectCat  <  5  Then 
MaintRx_01d  =  1 

N  u  m  Main  t_0 1  d_tc  m  p  =  NumMaint_01d_temp  +  1 
Else 

MaintRx_01d  =  0 
End  If 

.Cells(tempRow,  8)  =  MaintRx_01d 

If  DetectCat  >=  2  And  DetectCat  <  5  Then 
MaintRx_New  =  1 

NumMaint_New_temp  =  NumMaint_New_temp  +  1 
Else 

MaintRx_New  =  0 
End  If 

.Cells(tempRow,  10)  =  MaintRx_New 

'Continue  on  to  next  mission? 

If  SuccessMsn  =  0  Then 

.Cells(tempRow,  11)  =  "Baseline  Offline  until  PM" 
flagX  =  True 
End  If 

If  SuccessMsn2  =  0  Then 

.Cells(tempRow,  11)  =  "Sys  w/ISHM  Offline  until  PM" 
flagY  =  True 
End  If 

If  flagX  =  True  And  flagY  =  True  Then 
k  =  NumMissions  +  1 
End  If 

If  DetectCat  =  5  Or  TrueCat  =  5  Then 
k  =  NumMissions  +  1 
j  =  MaintNum  +  1 

.Cells(tempRow,  11)  =  "Catastrophic  Failure" 

End  If 

Next  k 

tempRow2  =  Range("A13").End(xlDown).Offset(l).Row 
.Cells(tempRow2,  1)  =  "End  of  Preventative  Maintenance  Cycle" 
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'Update  Metrics 

NumFalseAlarms  =  NumFalseAlarms  +  N  u  m  Fa  1  s  c  A 1  a  rm  s_tc  m  p 
NumFaultsDetected  =  NumFaultsDetected  +  NumFaultsDetected_temp 
NumFaults  =  NumFaults  +  NumFaults_tcmp 

NumlncorrectDeclared  =  NumIncDecl_temp  +  NumlncorrectDeclared 
NumSuccessMsns_01d  =  NumSuccessMsns_01d  + 

N  umS  ucces  sM  sns_01d_temp 

NumMaint_01d  =  NumMaint_01d  +  NumMaint_01d_temp 
NumSuccessMsns_New  =  NumSuccessMsns_New  + 
NumSuccessMsns_New_temp 

NumMaint_New  =  NumMaint_New  +  N  u  m  M  a  i  n  t_Nc  w_tc  m  p 
Next  j 

tempRow3  =  Range("A13").End(xlDown).Offset(l).Row 
.Cells(tempRow3,  1)  =  "End  of  Vehicle  Lifetime" 

Results(i,  1)  =  NumSuccessMsns_01d 
Results(i,  2)  =  NumMaint_01d 
Results(i,  3)  =  NumSuccessMsns_New 
Results(i,  4)  =  NumMaint_New 
Results(i,  5)  =  NumFaults 
Results(i,  6)  =  NumFaultsDetected 
Results(i,  7)  =  NumFalseAlarms 
Results(i,  8)  =  NumlncorrectDeclared 

Next  i 

End  With 

'Output  Results 

Worksheets("Results").  Activate 

With  Worksheets("Results") 

Application. Goto  .Range("Al:J30") 

For  i  =  1  To  MonteCarloNum 
.Cells(ll  +  i,  2)  =  i 

.Cells(ll  +  i,  3)  =  NumMissions  *  MaintNum 
For  j  =  1  To  8 

.Cells(l  1  +  i,  3  +  j)  =  Results(i,  j) 

SumTotal(j)  =  SumTotal(j)  +  Results(i,  j) 

Next  j 
Next  i 

.Cells(2,  5)  =  MonteCarloNum  *  NumMissions  *  MaintNum 
.Cells(3,  5)  =  SumTotal(l) 

.Cells(4,  5)  =  SumTotal(2) 
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.Cells(5,  5)  =  SumTotal(3) 

.Cells(6,  5)  =  SumTotal(4) 

End  With 

End  Sub 

Function  InputCheck()  As  Boolean 

flaglnput  =  True 

'Check  combobox  has  selections 
If  UserForml.SystemDist. Value  =  ""  Then 

MsgBox  "You  have  not  selected  a  distribution.  Please  select  a  distribution  and  " 
&_ 

"run  the  Monte  Carlo  simulation  again", ,  "Error" 
flaglnput  =  False 
Exit  Function 
End  If 

'Check  Parameter  inputs  are  valid  numbers 
If  IsNumeric(UserForml.SysParaml. Value)  =  False  Or 
IsNumeric(UserForml.SysParam2. Value)  =  False  Then 

MsgBox  "You  have  either  entered  a  non-numeric  value  for  the  system 
parameters"  &  _ 

"  or  left  a  field  blank.  Please  enter  a  numeric  value  and  run  the  Monte  Carlo 
simulation  again", ,  "Error" 
flaglnput  =  False 
Exit  Function 
End  If 

'Turn  parameter  inputs  into  numbers 
SysParaml  =  CDec(UserForml.SysParaml) 

SysParam2  =  CDec(UserForml.SysParam2) 

'Check  that  all  numbers  are  positive 
If  SysParaml  <  0  Or  SysParam2  <  0  Then 

MsgBox  "You  have  entered  in  a  negative  number,  all  numbers  should  be 
positive", ,  "Error" 
flaglnput  =  False 
Exit  Function 
End  If 

'Check  Failure  Properities  are  valid  numbers 
If  IsNumeric(UserForml.SysCatl. Value)  =  False  Or 
IsNurneric(UserForml.SysCat2.Value)  =  False  _ 
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Or  IsNumeric(UserForml.SysCat3.Value)  =  False  Or 
IsNumeric(UserForml.SysCat4.Value)  =  False  _ 

Or  IsNumeric(UserForml.SysCat5.Value)  =  False  Then 
MsgBox  "You  have  entered  a  non-numeric  value  for  a  probability  of  a  failure 
mode  occurance  or  have"  &  _ 

"  left  a  field  blank.  Please  enter  a  numeric  value  and  run  the  Monte  Carlo 
simulation  again", ,  "Error" 
flaglnput  =  False 
Exit  Function 
End  If 

'Turn  failure  inputs  into  numbers 
SysCatl  =  CDec(UserForml.SysCatl) 

SysCat2  =  CDec(UserForml.SysCat2)  +  SysCatl 
SysCat3  =  CDec(UserForml.SysCat3)  +  SysCat2 
SysCat4  =  CDec(UserForml.SysCat4)  +  SysCat3 
SysCat5  =  CDec(UserForml.SysCat5)  +  SysCat4 

'Check  Failure  Properties  are  between  0  and  1 

If  UserForml.SysCatl.Value  <  0  Or  UserForml. SysCatl. Value  >  1  Or 
UserForml.SysCat2. Value  <  0  Or  UserForml. SysCat2. Value  >  1  _ 

Or  UserForml. SysCat3. Value  <  0  Or  UserForml. SysCat3.Value  >  1  Or 
UserForml. SysCat4. Value  <  0  Or  UserForml. SysCat4. Value  >  1  _ 

Or  UserForml. SysCat5. Value  <  0  Or  UserForml. SysCat5. Value  >  1  Then 
MsgBox  "You  have  entered  a  failure  propability  greater  than  1  or  less  than  0." 
&_ 

"  Please  enter  a  correct  probability  and  run  the  Monte  Carlo  simulation 
again", ,  "Error" 
flaglnput  =  False 
Exit  Function 
End  If 

'Check  Failure  Properties  sum  to  1  for  each  system 
If  SysCat5  <>  1  Then 

MsgBox  "The  failure  properties  do  not  total  1.  Please  enter  a  correct  probability 
and  run  the  Monte  Carlo  simulation  again", ,  "Error" 
flaglnput  =  False 
Exit  Function 
End  If 

'Check  ISHM  Properities  are  valid  numbers 
If  IsNumeric(UserForml.ProbDetect. Value)  =  False  Or 
IsNumeric(UserForml.ProbFalseAlarm. Value)  =  False  Or 
IsNumeric(UserForml.ProbDiagnostic. Value)  =  False  Then 
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MsgBox  "You  have  entered  a  non-numeric  value  for  an  ISHM  property  or  left  a 
field  blank.  Please  enter  a  numeric  value  and  run  the  Monte  Carlo  simulation 
again", ,  "Error" 
flaglnput  =  False 
Exit  Function 
End  If 

'Check  ISHM  Properties  are  between  0  and  1 

If  UserForml.ProbDetect. Value  <  0  Or  UserForml.ProbFalseAlarm.Value  <  0  Or 
UserForml.ProbDiagnostic.Value  <  0  _ 

Or  UserForml.ProbDetect. Value  >  1  Or  UserForml.ProbFalseAlarm.Value 
>  1  Or  UserForml.ProbDiagnostic.Value  >  1  Then 

MsgBox  "You  have  entered  a  failure  propability  greater  than  1  or  less  than  0 
for  an  ISHM  property"  &  _ 

"  or  left  a  field  blank.  Please  enter  a  correct  probability  and  run  the  Monte 
Carlo  simulation  again", ,  "Error" 
flaglnput  =  False 
Exit  Function 
End  If 

'Turn  ISHM  inputs  into  numbers 
ProbDetect  =  CDec(UserForml.ProbDetect) 

Prob  Alarm  =  CDec(UserForml.ProbFalseAlarm) 

ProbDiagnostic  =  CDec(UserForml.ProbDiagnostic) 

'Check  Monte  Carlo  inputs  are  numerical 
If  IsNumeric(UserForml.MonteCarloNum.Value)  =  False  Or 
IsNumeric(UserForml.AverageMissionFength. Value)  =  False  _ 

Or  IsNumeric(UserForml.MaintFength. Value)  =  False  Or 
IsNumeric(UserForml. Fifetime. Value)  =  False  Then 

MsgBox  "You  have  either  entered  a  non-numeric  value  or  left  a  field  blank  in 
the  Monte  Carlo  frame.  "  &  _ 

"Please  enter  a  numeric  value.", ,  "Error" 
flaglnput  =  False 
Exit  Function 
End  If 

'Check  that  inputs  are  within  max  and  min  or  positive 
If  UserForml.MonteCarloNum.Value  <  0  Or 

UserForml.AverageMissionFength.  Value  <  0  Or  UserForml.  Fife  time.  Value  <  0 

Or  UserForml.MonteCarloNum.Value  >  500  Or 
UserForml. MaintFength. Value  <  0  Then 

MsgBox  "You  have  entered  a  negative  number  or  a  number  out  of  range  for 
Monte  Carlo  Simulations.  "  &  _ 
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"Please  enter  a  valid  number.", ,  "Error" 
flaglnput  =  False 
Exit  Function 
End  If 

'Turn  Monte  Carlo  Inputs  into  numbers 
MonteCarloNum  =  CDec(UserForml.MonteCarloNum) 

MissionLength  =  CDec(UserForml.AverageMissionLength) 

MaintLength  =  CDec(UserForml.MaintLength) 

Lifetime  =  CDec(UserForml.  Lifetime) 

NumMissions  =  Int(MaintLength  /  MissionLength)  'Rounds  down 
InputCheck  =  flaglnput 
End  Function 
Sub  ActivateStartPage() 

Worksheets("Intro").  Activate 
End  Sub 

Sub  ViewAssumptions() 

Worksheets(  "As  sumptions ")  .Activate 

Application. Goto  Worksheets("Assumptions").Range("Al :N30") 
ActiveWindow.Zoom  =  True 
Worksheets("Assumptions").Cells(l,  1). Select 

End  Sub 

Sub  SetUpConfusionMatrix() 

'Assumption  that  any  ISHM  diagnostic  algorithm  will  be  within  one  category  of 
the  true  category 

With  Worksheets("ConfusionMatrix") 

.Cells(2,  6)  =  ProbDiagnostic 
'Nominal  Column 
.Cells(8,  3)  =  ProbDiagnostic 
.Cells(9,  3)  =  1  -  ProbDiagnostic 
.Cells(10,  3)  =  0 
.Cells(ll,  3)  =  0 
.Cells(12,  3)  =  0 
.Cells(13,  3)  =  0 
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'Cat  1  Column 

.Cells(8,  4)  =  (1  -  ProbDiagnostic)  /  2 

.Cells(9,  4)  =  ProbDiagnostic 

.Cells(10,  4)  =  (1  -  ProbDiagnostic)  /  2 

.Cells(ll,  4)  =  0 

.Cells(12,  4)  =  0 

.Cells(13,  4)  =  0 

'Cat  2  Column 

.Cells(8,  5)  =  0 

.Cells(9,  5)  =  (1  -  ProbDiagnostic)  /  2 

.Cells(10,  5)  =  ProbDiagnostic 

.Cells(ll,  5)  =  (1  -  ProbDiagnostic)  /  2 

.Cells(12,  5)  =  0 

.Cells(13,  5)  =  0 

'Cat  3  Column 

.Cells(8,  6)  =  0 

.Cells(9,  6)  =  0 

.Cells(10,  6)  =  (1  -  ProbDiagnostic)  /  2 

.Cells(ll,  6)  =  ProbDiagnostic 

.Cells(12,  6)  =  (1  -  ProbDiagnostic)  /  2 

.Cells(13,  6)  =  0 

'Cat  4  Column 

.Cells(8,  7)  =  0 

.Cells(9,  7)  =  0 

.Cells(10,  7)  =  0 

.Cells(ll,  7)  =  (1  -  ProbDiagnostic)  /  2 

.Cells(12,  7)  =  ProbDiagnostic 

.Cells(13,  7)  =  (1  -  ProbDiagnostic)  /  2 

'Cat  5  Column 

.Cells(8,  8)  =  0 

.Cells(9,  8)  =  0 

.Cells(10,  8)  =  0 

.Cells(ll,  8)  =  0 

.Cells(12,  8)  =  1  -  ProbDiagnostic 
.Cells(13,  8)  =  ProbDiagnostic 

End  With 

End  Sub 

Sub  ReadConfusionMatrix() 

Dim  Temp(6)  As  Double 
flag  =  True 
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With  Worksheets("ConfusionMatrix") 

'Input  Check 
For  j  =  1  To  6 

Temp(j)  =  .Cells(15,  2  +  j).' Value 
If  Temp(])  <>  1  Then 

MsgBox  "The  columns  need  to  add  to  1,  please  reset  this  matrix", ,  "Error" 
flag  =  False 
j  =  7 
End  If 
Next  j 
End  With 

If  flag  =  False  Then 
Exit  Sub 
Else 

With  Worksheets("HiddenCM") 

For  i  =  1  To  6 
For  j  =  1  To  6 

ConfMatrix(i,  j)  =  .Cells(7  +  i,  2  +  j).' Value 
Next  j 
Next  i 
End  With 

Call  MonteCarloSim(True) 

End  If 

End  Sub 

Function  DeclareMatrix(RandCat  As  Double,  TrueCat  As  Single,  Detect  As  Single) 

If  Detect  =  0  Then 
DeclareMatrix  =  0 
Exit  Function 
End  If 

If  TrueCat  =  0  Then 
For  i  =  0  To  5 

If  RandCat  <  ConfMatrix(i  +  1,  1)  Then 
DeclareMatrix  =  i 
i  =  6 
End  If 
Next  i 

Elself  TrueCat  =  1  Then 
For  i  =  0  To  5 

If  RandCat  <  ConfMatrix(i  +  1,  2)  Then 
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DeclareMatrix  =  i 
i  =  6 
End  If 
Next  i 

Elself  TrueCat  =  2  Then 
For  i  =  0  To  5 

If  RandCat  <  ConfMatrix(i  +  1,  3)  Then 
DeclareMatrix  =  i 
i  =  6 
End  If 
Next  i 

Elself  TrueCat  =  3  Then 
For  i  =  0  To  5 

If  RandCat  <  ConfMatrix(i  +  1,  4)  Then 
DeclareMatrix  =  i 
i  =  6 
End  If 
Next  i 

Elself  TrueCat  =  4  Then 
For  i  =  0  To  5 

If  RandCat  <  ConfMatrix(i  +  1,  5)  Then 
DeclareMatrix  =  i 
i  =  6 
End  If 
Next  i 

Else 

For  i  =  0  To  5 

If  RandCat  <  ConfMatrix(i  +  1,  6)  Then 
DeclareMatrix  =  i 
i  =  6 
End  If 
Next  i 

End  If 

End  Function 

Sub  ClearWorkbook() 

With  Worksheets("Calculations") 

.Cells(4,  2).ClearContents 
.Cells(5,  2).ClearContents 
.Cells(6,  2).ClearContents 
.Cells(4,  5).ClearContents 
.Cells(5,  5).ClearContents 
.Cells(6,  5).ClearContents 
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.Cells(7,  5).ClearContents 
.Cells(8,  5).ClearContents 
.Cells(3,  ll).ClearContents 
.Cells(4,  ll).ClearContents 
.Cells(5,  ll).ClearContents 
.Cells(6,  ll).ClearContents 
.Cells(7,  ll).ClearContents 
.Cells(3,  14).ClearContents 
.Cells(4,  14).ClearContents 
.Cells(5,  14).ClearContents 
.Cells(6,  14).ClearContents 
.Cells(4,  16).ClearContents 
.Cells(4,  17).ClearContents 
.Cells(8,  16).ClearContents 
.Cells(8,  17).ClearContents 
.Range("al5:"  & 

.Range("al5").End(xlDown).End(xlToRight).  Address). ClearContents 
.Range("gl5:"  & 

.Range("gl  5  ").End(xlDown).End(xlToRight).  Address).  ClearContents 

End  With 

With  Worksheets("ConfusionMatrix") 

.Cells(2,  6). ClearContents 
.Range("c8:"  & 

.Range("c8").End(xlDown).End(xlToRight).  Address). ClearContents 

End  With 

With  Worksheets("Results") 

.Range( "E2 : "  &  .Range( "E2 ") .End(xlDown) .Address) .ClearContents 
.Range("bl2:"  & 

.Range("bl  2"). End(xlDown).End(xlToRight).  Address). ClearContents 

End  With 

Unload  UserForml 

Worksheets!  "Intro ")  .Activate 

End  Sub 

Sub  ViewCalcPage() 

W  orksheets(  "Calculations ")  .Activate 

End  Sub 


Sub  ViewResultsQ 
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Worksheets("Results").  Activate 
End  Sub 
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Appendix  D:  Design  of  Experiments  Results  and  Models 


Alias  Structure  with  Main  Effects  and  Low  Order  Interactions 

A  =  Weibull-Theta 
B  =  Weibull-Beta 
C  =  Sensor  Quality  (c) 

D  =  Probability  of  Detection  (PD) 

E  =  P(Fault  Category  II) 

F  =  P(Fault  Category  III) 

G  =  ISHM  Diagnostic  Confidence  Fevel  (Dcl) 


I  =  ABD  =  ACE  =  BCF  =  ABCG 


A  =  A  +  BD  +  CE  +  FG 
B  =  B  +  AD  +  CF  +  EG 
C  =  C  +AE  +  BF  +  DG 


D  =  D  +  AB  +  CG  +  EF 
E  =  E  +  AC  +  BG  +  DF 
F  =  F  +  BC  +  AG  +  DE 
G  =  G  +  CD  +  BE  +  AF 


Initial  Experiment  Results 

The  initial  experiment  results  can  be  found  below.  The  original  run  order  was  random; 
however,  this  data  has  been  sorted  to  place  the  center  points  at  the  bottom  for  easier 
analysis. 


131 


Run 

Response 

Weibull  - 

theta 

Weibull  - 

beta 

Sensor  Quality 

Prob 

Detection 

Prob  Fault 

Category  II 

Prob  Fault 

Category  III 

Diagnostic  CL 

1 

33 

-1 

-1 

1 

1 

-1 

1 

-1 

2 

21 

-1 

1 

1 

-1 

-1 

-1 

1 

3 

514 

-1 

-1 

-1 

1 

1 

-1 

1 

4 

368 

1 

1 

1 

1 

1 

-1 

-1 

5 

526 

1 

1 

-1 

1 

-1 

1 

1 

6 

97 

-1 

1 

-1 

-1 

1 

1 

-1 

7 

22 

1 

-1 

1 

-1 

1 

1 

1 

8 

613 

1 

-1 

-1 

-1 

-1 

-1 

-1 

9 

82 

-1 

-1 

1 

1 

-1 

1 

-1 

10 

98 

-1 

1 

1 

-1 

-1 

-1 

1 

11 

382 

-1 

-1 

-1 

1 

1 

-1 

1 

12 

487 

1 

1 

1 

1 

1 

-1 

-1 

13 

414 

1 

1 

-1 

1 

-1 

1 

1 

14 

144 

-1 

1 

-1 

-1 

1 

1 

-1 

15 

30 

1 

-1 

1 

-1 

1 

1 

1 

16 

166 

1 

-1 

-1 

-1 

-1 

-1 

-1 

17 

85 

-1 

-1 

1 

1 

-1 

1 

-1 

18 

23 

-1 

1 

1 

-1 

-1 

-1 

1 

19 

195 

-1 

-1 

-1 

1 

1 

-1 

1 

20 

330 

1 
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-1 
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The  response  for  each  run  is  the  total  of  the  four  repeated  measurements,  as  seen  below: 


RM-  1 

RM  -  2 

RM  -  3 

RM  -  4 

Response 

2 

6 

0 

25 

33 

0 

2 

15 

4 

21 

13 

324 

53 

124 

514 

0 

49 

148 

171 

368 

213 

169 

100 

44 

526 

10 

3 

1 

83 

97 

15 

5 

2 

0 

22 

242 

10 

69 

292 

613 

21 

60 

1 

0 

82 

3 

23 

1 

71 

98 

92 

124 

11 

155 

382 

45 

1 

344 

97 

487 

44 

163 

51 

156 

414 

1 

27 

108 

8 

144 

5 

9 

13 

3 

30 

29 

0 

17 

120 

166 

10 

26 

2 

47 

85 

2 

8 

4 

9 

23 

0 

50 

106 

39 

195 

213 

109 

2 

6 

330 

149 

222 

94 

356 

821 

36 

6 

2 

6 

50 

13 

4 

7 

1 

25 

45 

91 

90 

83 

309 

6 

3 

4 

96 

109 

48 

16 

8 

54 

126 

13 

1 

0 

7 

21 

3 

3 

1 

4 

11 

Initial  Model  Analysis 

Using  these  results,  the  sum  of  squares  for  the  factors  and  their  interactions  were 
calculated  and  it  was  found  that  Weibull-Theta,  Sensor  Quality,  and  PD  were  significant. 
Due  to  aliasing,  the  sum  of  squares  for  Pd  also  includes  the  interaction  between  Weibull- 
Theta  and  Weibull-Beta;  however,  since  Weibull-Beta  is  not  significant  to  the  model,  it 
can  be  assumed  that  Pd  is  the  true  significant  factor. 
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Regressing  these  factors  against  the  test  data  gave  a  fairly  high  coefficient  of 
determination  (R2)  value  of  0.704  and  an  Fo  value  of  13.67  (p-value  of  less  than  0.0001), 
as  seen  below.  These  values  signified  that  the  model  explained  most  of  the  variability  in 
the  data,  and  further  exploration  of  the  individual  factors  confirmed  that  they  all  still 
significantly  contributed  to  the  model. 


A  Analysis  of  Variance 


Source 

DF 

Sum  of 
Squares 

Mean  Square 

Model 

4 

921790.6 

230448 

Error 

23 

387551.2 

16850 

C.  Total 

27 

1309341.9 

A  Summary  of  Fit 

RSquare 

0.704011 

F  Ratio 

RSquare  Adj 

0.652534 

13.6764 

Root  Mean  Square  Error 

129.8078 

Prob  >  F 

Mean  of  Response 

217.9286 

<0001* 

Observations  (or  Sum  Wgts) 

28 

Examination  of  the  model  residuals  did  not  indicate  any  normality  assumptions  were 
violated  (no  apparent  pattern  or  significant  tailing  in  the  variance  checks).  The  Residual 
by  Predicted  Plot  and  the  Normal  Quantile  Plot  can  be  seen  below. 
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Variability  of  the  non-significant  factors  was  also  investigated,  with  the  following 
settings  to  be  determined  as  causing  less  variance  in  the  results. 


Factor 

Best  Setting  for 
Variability 

Weibull  -  Beta 

Either 

P(Fault  Category  II) 

High  -  0.4 

P(Fault  Category  III) 

Either 

Dcl 

Either 

The  Center  Points  that  were  chosen  indicated  that  curvature  is  present,  as  seen  by  a  p- 
value  under  0.05  below.  This  means  the  assumption  of  linearity  in  the  factor  effects 
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cannot  be  maintained  and  axial  points  need  to  be  added  for  further  analysis.  Curvature 
can  be  investigated  through  a  central  composite  design. 


a  Parameter  Estimates 


Term 

Estimate 

Std  Error 

t  Ratio 

Prob>|t| 

Intercept 

243.125 

26.4969 

9.18 

<0001* 

Weibull  -  theta 

99.458333 

26.4969 

3.75 

0.0010* 

Sensor  Quality 

-109.4583 

26.4969 

-4.13 

0.0004* 

Prob  Detection 

109.95833 

26.4969 

4.15 

0.0004* 

Curvature 

-176.375 

70.1042 

-2.52 

0.0193* 

Central  Composite  Model 

Axial  points  were  then  added  to  the  test  design  to  further  explore  the  response  surface. 

For  a  3-factor  experiment,  an  axial  point  of  3  (0.316)  will  be  used  to  ensure  the  design 

is  fully  rotatable.  The  new  test  design  settings  in  natural  units  can  be  seen  in  the  table 
below.  The  remaining  test  factors  will  be  kept  at  their  low  settings  except  for  P(Fault 
Category  II),  which  will  be  kept  high.  The  axial  points  will  be  tested  in  random  order 
with  two  replications  for  a  total  of  18  additional  test  points  and  will  be  added  the  previous 
test  results  for  re-analysis. 


Factor 

Axial  (-1.316) 

Axial  (1.316) 

Weibull-Theta 

652 

1047 

Sensor  Quality 

36.8 

563 

Pd 

0.21 

0.99 

The  second  experiment  results  can  be  seen  below: 
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Response 

Weibull  - 

theta 

Weibull  - 

beta 

Sensor  Quality 

Prob 

Detection 

Prob  Fault 

Category  II 

Prob  Fault 

Category  III 

Diagnostic  CL 

460 

1.316 

-1 

0 

0 

1 

-1 

-1 

143 

-1.316 

-1 

0 

0 

1 

-1 

-1 

847 

0 

-1 

0 

1.316 

1 

-1 

-1 

106 

0 

-1 

0 

-1.316 

1 

-1 

-1 

1186 

0 

-1 

1.316 

0 

1 

-1 

-1 

63 

0 

-1 

-1.316 

0 

1 

-1 

-1 

197 

1.316 

-1 

0 

0 

1 

-1 

-1 

55 

-1.316 

-1 

0 

0 

1 

-1 

-1 

586 

0 

-1 

0 

1.316 

1 

-1 

-1 

0 

0 

-1 

0 

-1.316 

1 

-1 

-1 

824 

0 

-1 

1.316 

0 

1 

-1 

-1 

33 

0 

-1 

-1.316 

0 

1 

-1 

-1 

64 

1.316 

-1 

0 

0 

1 

-1 

-1 

167 

-1.316 

-1 

0 

0 

1 

-1 

-1 

776 

0 

-1 

0 

1.316 

1 

-1 

-1 

5 

0 

-1 

0 

-1.316 

1 

-1 

-1 

1498 

0 

-1 

1.316 

0 

1 

-1 

-1 

40 

0 

-1 

-1.316 

0 

1 

-1 

-1 

As  expected  from  the  results  of  the  curvature  analysis,  a  second-order  response  surface 
was  discovered.  There  were  also  some  low-order  interactions  and  a  main  effect  (P(Fault 
Category  III))  that  were  found  to  be  newly  significant.  However,  there  is  clear  funneling 
in  the  residuals,  indicating  that  a  transformation  of  the  response  variable  is  necessary. 
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Since  the  funneling  is  consistent  with  a  Poisson  random  variable  (the  variance  of  y  is 
proportional  to  the  regressor),  a  transformation  of  y  to  Jy  is  appropriate.  With  this 
transformation,  the  residuals  retained  a  scatter  pattern  and  did  not  indicate  any  normality 
assumptions  were  violated.  The  new  Residual  by  Predicted  Plot  and  the  Normal  Quantile 
Plot  can  be  seen  below. 
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The  final  coded  parameter  estimates  and  significant  factors  and  interactions  can  be  seen 
below: 


21  Parameter  Estimates 


Term 

Estimate 

Std  Error 

t  Ratio 

Prob>|t| 

Intercept 

6.0184269 

2.586194 

2.33 

0.0255* 

Prob  Detection 

5.3295735 

0.762556 

6.99 

<0001* 

Prob  Fault  Category  III 

-8.842107 

2.019323 

-4.38 

<0001* 

Weibull  -  theta 

2.5606553 

0.762556 

3.36 

0.0018* 

Sensor  Quality 

1.6622854 

0.776798 

2.14 

0.0390* 

(Prob  Fault  Category  lll+0.3913)*(Prob  Fault  Category  111+0.3913) 

8.9648706 

2.594351 

3.46 

0.0014* 

Prob  Detection*Weibull -theta 

-13.09048 

1.895008 

-6.91 

<0001* 

Weibull  -  theta'Weibull  -  theta 

-2.859716 

1.226968 

-2.33 

0.0253* 

(Prob  Fault  Category  lll+0.3913)*Sensor  Quality 

-14.29718 

1.660662 

-8.61 

<0001* 

Regressing  these  factors  against  the  test  data  gave  a  higher  coefficient  of  determination 
(R2)  value  of  0.81  and  an  F0  value  of  16.73  (p-value  of  less  than  0.0001)  than  the  initial 
model,  as  seen  below.  This  new  model  also  gives  a  significantly  smaller  Mean  Square 
Error,  another  indication  that  this  model  is  a  better  fit  to  the  data  than  the  initial  model. 


j  Analysis  of  Variance  j  Summary  of  Fit 


Sum  of 

RSquare 

0.812099 

Source 

DF 

Squares 

Mean  Square 

F  Ratio 

RSquare  Adj 

0.771471 

Model 

8 

3085.2201 

385.653 

16.7349 

Root  Mean  Square  Error 

4.471932 

Error 

37 

852.6568 

23.045 

Prob  >  F 

Mean  of  Response 

14.15299 

C.  Total 

45 

3937.8769 

<0001* 

Observations  (or  Sum  Wgts) 

46 

The  final  model  equation  mapping  the  response  surface  in  natural  values  is  seen  below: 
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a  Prediction  Expression 


-5.8606757513194 
+0.01 544469484738*Weibull -theta 
+0.00  7  4  8  6  47  1  27854*Sensor  Quality 
+  17.3222863745496*Prob  Detection 


+-67.369395787959*Prob  Fault  Category  III 


+ 


+ 


(Prob  Fault  Category  111-0.19130434782609) 
.((Prob  Fault  Category  111-0.19130434782609) 
1*501.71480745745 
(Weibull  -  theta-849. 9999902 1739l) 

,[(Prob  Detection-0.65217391304348)  1 
(*-0.237351067408  J 

(Weibull -theta-849.999990217391) 
.((Weibull  -  theta-849.999990217391) 
(*-0.0001395707596 

(Prob  Fault  Category  111-0.19130434782609) 
.[(Sensor  Quality-300.000013043478)  ) 


*-0.4148878097853 


\2 


From  this  equation  the  stationary  point  is  a  point  of  minimum  response,  clearly  visible  in 
the  figure  below. 


prob  Detection 


Weibull  -  theta 


We*ull  -  theta 


While  this  response  surface  best  illustrates  the  region  were  the  response  is  at  its 
minimum,  or  the  region  where  adding  ISHM  to  the  UAS  baseline  would  not  significantly 
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affect  mission  success  rates,  there  can  be  some  inferences  made  about  the  regions  that 
maximize  the  response.  By  determining  the  sensor  properties  significant  and  not  the  only 
ISHM  performance  characteristic  (the  Diagnostic  Confidence  Level),  this  evaluation 
implies  that  the  benefits  or  disadvantages  of  adding  ISHM  rely  primarily  on  the 
performance  of  the  baseline  health  management  system.  Specifically,  that  ISHM  becomes 
more  beneficial  as  the  baseline  health  management  system  performs  worse. 

Based  on  these  results,  when  evaluating  whether  to  add  ISHM  to  a  vehicle,  decision 
makers  should  compare  the  cost  and  mission  benefits  of  upgrading  just  the  sensors  with 
the  cost  and  mission  benefits  of  adding  ISHM.  The  two  options  have  roughly  equivalent 
installation  labor  costs,  as  each  would  have  to  be  implemented  at  the  subsystem  level  (the 
difference  being  replacing  the  sensors  versus  adding  a  module  to  an  existing  sensor),  so 
the  main  comparison  would  be  the  cost  of  the  new  sensors  versus  the  cost  of  the  ISHM 
technology  weighed  against  the  difference  in  expected  mission  success  rates. 
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management  technology  and  ISHM.  Using  representative  data  for  a  UAS.  the  analysis  determined  that  the  failure  distribution  parameters, 
sensor  quality  (which  determines  the  relationship  between  probability  of  detection  and  probability  of  false  alarm),  and  probability  of  an 
imminent  fault  during  a  mission  were  significant  to  the  model.  The  result  of  the  model  determined  that  ISHM  can  result  in  a  significant 
improvement  on  mission  assurance,  especially  when  implemented  with  higher  quality  sensors  and  on  vehicles  where  the  probability  of 
imminent  failure  is  higher  relative  to  the  mission  times  and  time  between  preventative  maintenance.  This  appears  consistent  with  the 
premise  that  ISHM  can  support  an  extension  of  preventative  maintenance  intervals  with  an  attendant  reduction  in  sustainment  cost. 
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